VMware Certified Professional Data Center Virtualisation VCP6-DCV
Time to renew my VCP qualification, no time to lose, only got barely 3 months to do it! This article contains the technical lab and notes that I have taken to study and renew my VMware Certified Professional Data Center Virtualisation VCP6-DCV. Hope it helps you in anyway, practise and revise hard!
Passed 25th August 2017, 410 out of 500 (82%), oh yeath, what a relief! :)
Step1: Installing vCenter
Installing VCSA (vCenter Server Appliance) is actually a nightmare with version 6.0, have not tested with 6.5 but have heard there are mayor improvements, let's hope so.
- Before installing the VMware Client Integration Plug-in 6.0, be sure to turn off your antivirus! The program will actually modify your hosts file, and most antivirus solutions will block that I managed to install the Plug-in without issues in a Win 2008 R2 with IE11 version 11.0.9600.17207
- Needless to say, you need to right-click on the Plug-in and choose to install with admin rights
- Ensure you set the IE11 to "Automatically detect intranet network"
vCenter/vsphere tips:
- To check the proper certificate of the ESXi host server, visit the "View Support Information" in the DCUI
- VMware does not support concurent deployments of PSC in the same SSO domain
- Enhanced Link Mode is only available from Standard up license (not foundation or essentials)
- Shares are ignore ift there is no contention
- Enterprise Plus license is required for Network IO control
- Cross-vCenter vMotion only comes with Enterprise Plus. Enterprise+ also includes Storage DRS, Storage IO Control (SIOC), Single Rool IO Virtualizacion (SR-IOV), Network IO Control (NIOC), Flash Read Cache (vFRC), vDS, Host Profiles, Auto Deploy
VMCA (VMware Certificate Authority)
- VMCA does not support CRLs
- VMCA does not have the concept of certificate revocation, you'll have to replace all
- VECS (VMware End Point Certificate Store) is a local repository for certificates and private keys VECS does not store ESXi host certificates, those are locally in /etc/vmware/ssl folder
VMCA (VMware Certification Authority), runs in the PSC, responsible for issuing certificates for solution users, machine certificates and ESXi certificates
VECS (VMware Endpoint Certificate Storage) is a local repository for certificates and private keys
ESXi certificates are stored locally on the ESXi hosts "etc/vmware/ssl" locationvecs-
Ensure that when you export VM, they have no DVD/CD attached
- OVF (Open Virtualisation Format), set of files
- OVA (Open Virtualisation A ), all in one file
Multipath iSCSI (two nic) and use separeate physical network (if not, defenitely separeate VLANs). Also, iSCSI uses no routing at all, so ensure all stays in the same VLAN
Do multipath for iSCSI
New stuff to know for vSphere 6
Harden virtual machines
- isolation.tools.diskWiper.disable = TRUE
- isolation.tools.diskShrink.disable = TRUE
- isolation.tools.copy.disable =TRUE
- isolation.tools.paste.disable = TRUE
- isolation.device.connectable.disable = TRUE
- isolation.device.edit.disable = TRUE
- vmx.log.keepOld = 10 //to disable logging all together, add the following line instead: logging = FALSE
- tools.setInfo.sizeLimit = "2000000" /set the size of the VMX file to 2MB instead of the default 1MB, so more info can be added
- tool.guestlib.enableHostInfo = FALSE /PerfMon counter disabled by default
- These are the unexposed features:
- isolation.tools.unity.push.update.disable -TRUE
- isolation.tools.ghi.launchmenu.change =TRUE
- isolation.tools.ghi.autologon.disable =TRUE
- isolation.tools.hgfsServerSet.disable =TRUE
- isolation.tools.memeSchedFakeSampleState.disable -TRUE
- isolation.tools.getCreds.disable =TRUE
The following 2 lines are added by default, but ensure they're there otherwise VMTools can be use to eject devices
- isolation.device.connectable.disable = TRUE
- isolation.device.edit.disable = TRUE
FT and HA are not supportetd for VMs that have 3D graphics enable
If the machine is slow you can disable Accelerated 3D Graphics or add the following to the VMX file:
- vga.vgaOly = TRUE
If the VMware Tools installation hangs on a VM, use this command on the host to cancel it:
- vim-cmd vmsvc/tools.cancelinstall "vm.id"
VMware tools in Linux use "sudo apt-get install open-vm-tools"
If when typing characters are repeated, add this line to the VMX file, increasing the threshold for auto-repeat
- keyboard.typematicMinDelay = "2000000"
To use FT Legacy on a VM, add this entry:
- vm.uselegacyft = TRUE
Harden ESXi host
- logging = FALSE //add this line to the file /etc/VMware/config to disable VM logging
- Security.PasswordQualityControl = control the strengh of the DCUI password
- min=disabled,disabled,disabled,7,7
- min character accepted for 1 class, min accepted for 2 classes, min accepted for passphrases, min accepted for 3 classes (in the example above 7), min accepted for 3 classes plus passphared, meaning 4 classes
- Uppercase at begging and number at the end don't count
- MOB (Managed Object Browser) is disable by default to prevent attacks, but you have to enable it if you want to extract an old certificate from the host
- Config.HostAgent.plugins.solo.enableMob = FALSE
You can control the behavious of SSH by modifying /etc/ssh/sshd_config file, the paremeter PermitRoolLogin
Net.IOControlPnicOptOut = vmnic1, vmnic2, etc //exclude the nic for participating in NIOC
Harden vCenter Server
The following filters are by default set to TRUE (enable) so that vCenter cannot detect the present of the related object when scanning for storage
- For RDM objects = config.vpxd.filter.rdmFilter
- For VMFS objects = config.vpxd.filter.vmfsFilter
- For Host Rescan = config.vpxd.filter.hostRescanFilter
- For Same Host and Transports = config.vpxd.filter.SameHostAndTransportsFilter
To setup MSCS (Microsoft Cluster) the RDM filter must the set to FALSE, so that vCenter can detect the LUN and add it as an RDM even though is being use as an RDM by another VM
If the performance graphics are not displaying correctly, or if the statistics collection level is higher than 1, you many need to truncate these two tables in the vCenter database:
- truncate table VPX_HIST_STAT1
- truncate table VPX_SAMPLE_TIME1
There is a script that you can donwload from KB2110031 to reduce the amount of historical data on the PostgressSQL database
#service-control --stop --all ;stops all vCenter services, use then the --start to initiate then
To remove the warning about management network redundancy, set this on the advance options:
- das.ignoreRedundantNetWarning = true
ESXi Networking
Load-based teaming is only offered in vDS. Standard switches only offer:
- Originating virtual port ID
- Source MAC address
- Source and Destination IP hash
Dynamic binding = NO, it's been decreciated from vSphere 5.0; dynamic binding means the adapter is only connected to the vDS when the machine is on
Ephemeral = YES, ports are created and deleted on demand, just like Elastic
When LACP is in used, you cannot configure port mirroning
Remember that a dvPort group can be used both by a VM and by a VMkernel port
PVLANs need to be configure at the vDS level .
- Isolated PVLAN, only talk to the promiscuous
- Community PVLAN, talk to other communities as well as the promiscuos
TSO (TCP Segmentation Offload) is enable by default, and if you change it reverts to its default after a reboot. To change it permanentely the command needs to be added to /etc/rc.local/local.sh
- To identify whether it's enable or disable run: esxcli system settings advanced list -o /Net/UseHwTSO (if it says 1 is enable)
- To disable TSO in Linux run #ethtool -K vmnic# tso off
- To enable TSO in Windows add this line to .vmx: enthernet#.features = "0x2"
Network recovery (rollback) is not suported on stateless configuration of auto deploy hosts (with the configuration installed in RAM)
ESXi firewall commands: https://pubs.vmware.com/vsphere-51/index.jsp?topic=%2Fcom.vmware.vsphere.security.doc%2FGUID-7A8BEFC8-BF86-49B5-AE2D-E400AAD81BA3.html
VMware Knowledge base - useful articles
How to add the certificates of vCenter for all green approval https://kb.vmware.com/kb/2108294
Virtual Machines Hardware Versions https://kb.vmware.com/kb/1003746 Here a list of newly added features of HW version 11 http://www.running-system.com/vsphere-6-new-virtual-hardware-version-11-vhw11/
Determine your build version: https://kb.vmware.com/kb/1022196 , then check your build numbers against this list https://kb.vmware.com/kb/1014508
Editing files on an ESX host using vi or nano https://kb.vmware.com/kb/1020302
Integration Plug-in fails to install: https://kb.vmware.com/kb/2130672 Rubbish, use the flat client to deploy ova for the time being
System Logs are stored on non-persistant storage https://kb.vmware.com/kb/2032823
Upgrading from vSphere 5.x to 6 https://kb.vmware.com/kb/2057795
Backup and Restore the vPosgree database https://kb.vmware.com/kb/2091961
Restart the ESXi host management agents through CLI https://kb.vmware.com/kb/1003490
Understanding network rollback https://kb.vmware.com/kb/2032908
All TCP and UDP ports use in the vSphere Universe: https://kb.vmware.com/kb/1012382
After a Windows installation of vCenter, you have NO ACCESS AT ALL after logging in: You need to override the path as stated in here: https://communities.vmware.com/thread/507933
Please try the following. (before that please create a backup from the vCenter server) In regedit system wide path is defined here: Computer\HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\Environment->Path Local system account overridden Path is defined under: Computer->HKEY_USERS->S-1-5-18\Environment->Path (first step, second location can be verified if exists; and if exists values can be compared to see differences) You can either remove(rename) the existing override (Computer->HKEY_USERS->S-1-5-18\Environment->Path) completely. This will make the system wide Path to take effect. Or if this override was specified on purpose (for some reason), then modify Computer->HKEY_USERS->S-1-5-18\Environment->Path to make sure to include MIT Kerberos installation (such as c:\Program Files\MIT\Kerberos\bin), and possibly other vSphere paths (like OpenSSL) for completeness.
VMware ESXi 6 Password Policy: https://www.ivobeerens.nl/2015/10/07/vmware-esxi6-password-policy/
Configure vFlash Read Cache (vFRC) https://www.vladan.fr/vmware-vflash-read-cache-vfrc/
Visit the VMware Labs (flings) to keep an eye in future develops: https://labs.vmware.com/flings/
Data Collections Levels: https://pubs.vmware.com/vsphere-50/index.jsp?topic=%2Fcom.vmware.vsphere.monitoring.doc_50%2FGUID-25800DE4-68E5-41CC-82D9-8811E27924BC.html
Configuration: https://docs.vmware.com/en/VMware-vSphere/6.0/vsphere-esxi-vcenter-server-601-appliance-configuration-guide.pdf
All TCP/UDP ports used in the vSphere universe: https://kb.vmware.com/kb/1012382
Resources
The 5 states of ESXi hosts are based on the "minFree" value, which is 899MB of the first 28GB of the host, plus 1% of any additional memory; the states are:
- High - equals 400% of minFree
- Clear - equals 100% of minFree
- Soft - 64%
- Hard - 32%
- Low - 16%
This is what the host does at when the threshold at the different stages is crossed:
- When below High (< 400%), large pages are broken into 4KB pages
- When below Clear, TSP is actively called instead of waiting for the next schedule TSP run
- When below Soft, balloning begins
- When below Hard, compression and swapping begins
- At Low, blocking begins, and certain VMs are preventing from allocating memory
The mechanism:
- Transparent Page Sharing (TSP); identify identical pages and move them into a single file; intra-VM TPS is enable by default (but for security inter-VM TPS is disable
- Balloning; memory pages used by the VM are reclaimed by the host and given to another VM; VMware Tools ballon driver (vmemclt) is used for that. The host will only claim up to 65% of the VM memory, but this can be change in the VM vmx file Mem.ctlMaxPercent
- Compression is a sign of contention, it only engages at hard and low
- Swapping move ram from the physical ram to the disk, two types of swap can happens:
- Guest OS swap; inside the guest, the famout pagefile.sys, this can occur at any time
- VM Swapping; the VMkernel moves ram to the .vswp file of the VM, this happens at Hard and Low
- Memory Overhead; the amount of ram the VMkernel uses to actually run the VM
- Memory Consumed = assigned mem + overhead
Storage Metrics:
- Physical Device Latency; should be less than 10 milliseconds, the time the device takes to process iSCSI commands
- Kernel Latency; should be less than 1 millisecond, the time VMkernel process commands from VM to storage
- Guest Latency; it should be less than 15 milliseconds
vCenter tips
Cluster HA
To define additional isolation addresses for HA to ping to, add the advance command das.isolationaddress0 ;you can configure as well this setting: das.usedefaultisolationaddress
HA requires the following ports to be open
- Inbound TCP and UDP ports 8042 to 8045
- Outboud TCP and UDP ports 2050 to 2250
If HA takes a while to be configure on the host, it may timeout, if it does please extend the timeout periof of HA by adding this advance command to a value greater than the default of 240 seconds vpxd.das.electionWaitTimeSec
To increase the amount of time HA wait for a VM to shutdown after the power off command has been given, add a value greater than 300 seconds to the advance command das.isolationShutdownPeriod
To prevent false positive increase the grace period for network isolation here: fdm.isolationpolicydelaySec
HA uses a feature called Fault Domain Manager (FDM), and therefore it is not dependeable of vCenter to work, e.g. vCenter server can be down and HA will still work
If you don't have enough storage for the heartbeats, you can disable the warning by seeting das.ignoreInsufficientHbDatastore to true
HA, unlike DRS, does not respect anti-affinity rules unless: das.respectvmvmantiaffinityrules is set to true (default is false)
Here you can find advance settings for HA: https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2033250
DRS
Requires Enterprise edition
vSAN
Check this out for the default vSAN policy: https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.virtualsan.doc/GUID-C228168F-6807-4C2A-9D74-E584CAF49A2A.html
To access the VAMI, typ the port 5480 on the vCenter and logon as root: https://myvcenter.local:5480
The PSC (Platform Service Controller) GUI: https://myvcenter.local/psc
The Applicance Support Bundle (in case VC is unavailable): https://mycenter.local/applicance/support-bundle
VCSA typical CLI commands (applicanesh)
- From a esxi host ssh command line, you can go to the vCenter by typing: # ssh root@myVcenterIPAddress
- #df -h ; check the hard drive capacity of vCenter
- #vimtop ;similar to esxtop
Fault Tolerance
Legacy FT and the new version of FT (Symetric Multi-Processing) called SMP-FT can coexist side by side
- SMP-FT needs a 10Gbit network
- Enterprise Plus is required to protect VM with 4 x vCPUs
AutoDeploy
It needs the following components
- DHCP Server
- PXE boot enable
- TFTP server for the images
- vCenter for the host profiles
- Auto Deploy server for rules engines
- Answer file (those are managed only through the vSphere Client, and not the web client)
A stateless installation; the hosts needs auto deploy to install on every single boot, thouhg it catches the install on USB drive, etc
A stateful installation; the host only needs auto deploy for the 1st boot, there after run from local installation
To connet to vCenter user PowerCLI run:
Get-ExecutionPolicy, and if needed Set-ExecutionPolicy Unrestricted
Connect-VIServer vcsa6.nazaudy.internal
CLI commands and Storage theory
For the ESXi hosts, a list of useful Linux commands:
#cat /etc/hosts ; shows the content of the hosts file
#cp /etc/hosts /var/tmp ;copy the file hosts to the location var > tmp
#vi /etc/hosts ;open the file with the vi text editor
- Press Shit+Alt to start editing in a line above the cursor
- Press "i" to start editing where the cursor is
- Press "o" to start editiing in a line below the cursor
- Press the letter "d" twice to delete a whole line
- ESC + :q' > exit without saving
- ESC + :wq > save changes, if you get error that file is read only, add ! at the end (:wq!)
When logging on to the vCenter by ssh, run this command to enable the shell
- Command> shell.set --enable true
- Command> shell
- vol#
To fix problems converting host drives from SSD to HDD run: http://techhead.co/cannot-change-the-host-configuration-error-message-when-adding-disk-storage-to-a-vmware-vsphere-esxi-host/
- #ls- lha /vmfs/devices/disks
- #partedUtil getptbl /vmfs/devices/disks/vml.0000000000766d686261313a333a30
- #partedUtil setptbl /vmfs/devices/disk/vml.0000000000766d686261313a333a30 msdos
To shutdown a virtual machine using ssh from the host where it is running http://nigelhickey.com/power-vm-via-ssh/
- #esxcli vm process list
- #esxcli vm process kill -t [soft,hard,force] -w WorldNumber
To reboot a host:
- esxcli system maintenanceMode set --enable true
- esxcli system
Storage
iSCSI Software Initiator is the only one that support biridectional CHAP
Designate separates network adapters for iSCSI for performance and security
VSAN does not support IPv6
NFS4.1 is not compatible with SDRS, SIOC, SRM and VVOLs
NFS4.1 native multipathing for nic teaming policy is IP Hash; NFS4.1 is a big improvement in vSphere6, as before only 1 x IP was use to connect to a NFS share, now you can use Session Trunking and Multipahting
VASA (vSphere APIs for Storage Awareness), the storage send info about health, performace, etc to the vCenter; each vendor can setup its VASA provider differently
PSA (Pluggable Storage Architecture) configures and manages multipathing failover, it has the task of assigning to each storage device a MPP (Multi-Path Plugin) by using pre-define rules. There can be 2 types of MPP:
- NMP (Native Mutipathing Plugin), provided by VMware, it contains the following:
- MRU (Most Recently Used) - VMW_PSP_MRU ;by default MRU is also selected for ALUA devices; this is the default for active/passive arrays
- Fixed - VMW_PSP_FIXED ;it's the default to active/active arrays, iSCSI and Fibre Channel
- Round Robin - VMW_PSP_RR; the default use for active/active arrays when multipathing is involved
- SATP (Storage Array Type Plug), created by VMware for every array on the HCL
- PSP (Path Selection Policy), it selects which physical path to use for storage transport. VMware has 3 built-in PSPs:
- Third-Party MPP, supplied by the vendor storage
More info about multipathing considerations here: https://pubs.vmware.com/vsphere-51/index.jsp?topic=%2Fcom.vmware.vsphere.storage.doc%2FGUID-4D64F3DA-9701-4210-B34A-0A44D3A0100C.html and here too: https://docs.vmware.com/en/VMware-vSphere/6.0/com.vmware.vsphere.storage.doc/GUID-C1C4A725-8BE4-4875-919E-693812961366.html
VAAI (vSphere APIs for Array Integration), first introduced in 4.1; is a method for offloading operations from the host to the array. VAAI has 3 main capabilities, called primitives:
- Full Copy; the scsi extended copy command is replaced by the XCOPY command provided by VAAI
- Block Zeroing
- Hardware-assisted locking
VMCP (Virtual Machine Component Protection), it protects VMs when APD or PDL occurs
- APD (All Paths Down), defaults to 140 seconds
- PDL (Permanent Device Lost, added from vSphere5.0), after 140 seconds of APD the host stop sending I/O requests
VAAI - Storage APIs for Array Integration
Another improvement in vSphere6 is Bus Sharing: now you can set the HDD on a VM to be share with others
When "multi-writer locking" is enable, a VMDK is accessible by several computers, perfect for Microsoft Clustering
vFlash Read Cache (vFRC)
- Act as tier between the VM and the Storage
- Introduced in vSphere 5.5
- The host needs to have at least 1 x SSD
- Enterprise Plus license accepted only
- The VM needs to be at least hardware version 10 (5.5 compatible)
Storage commands
#esxcli storage core adapter list ;list all storage drives
Any device to which you want to change the PSP, needs to have its path unclaim and then reclaim, for example the below command changes the default MRU to Round Robin, remember that after the change the host needs to be rebooted:
#esxcli storage nmp satp set -s VMW_SATP_CX -P VMW_PSP_RR
VASA = Awareness
VMware mosaic of appliances
vRealize Orchestator, how to access it
After you import the "VMware-vRO-Appliance-7.0.1.17606-3571217_OVF10.ova" file, open the console and take note of the Orchestator Server IP address and port:
Then visit that address and click on "Start Orchestrator Client"
Open the aplication with Java, as prompted, and logon with the details "vcoadmin" and "vcoadmin"
You can also visit https://10.10.10.25:8283/vco-config/ and use "vmware" and "vmware" OR root and your own password
vRealize Operations, how to access it
Start by configuring all to DHCP, otherwise you may find this error about ./install.sh KB2150424
After the deploy, reduce the RAM size to 8GB or something that accomodates your host
It uses a few databases, Global xDB and File System Database (FSDB)
A vRealize Ops Manager cluster can contain multiple nodes:
- Master
- Master Replica
- Data
- Remote Collector
The 3 architectures components of vROps are:
- Administrative Server
- Analytics Server
- Database Server
VDP (VMware Data Protection)
- Visit https://10.10.10.90:8543/vdp-configure after installation
- Initial password: changeme
- Make sure time is set to: Europe\London
- Run the installation wizard and attach to the vCenter
- Use the vCenter plugin to connect to the VDP
NOTE: After pwoering on a recovered Windows VM running PSC, do not restart the server until you run the psc-restore script
VRA (vSphere Replicaton Appliance)
First of all import the OVF "vSphere_Replication_Server_SRM_OVF10" into vCenter, that will create the vRA Server from which you'll replicate to a remote site
After you logon, visit System > Time Zone and make sure time is set to: Europe\London
At the remote site, import hte vRA_AddOn_OVF10.ovf, and then register to it the vRA Server
vSphere Replication does not support VSS quiescing on VMs
vMA (Management Assistant)
The username for the vMA is "vi-admin"
How to add the vCenter and ESXi host to the vMA: https://pubs.vmware.com/vsphere-50/index.jsp?topic=%2Fcom.vmware.vma.doc_50%2Fvima_get_start.4.12.html
Basicallly, you need to enter the command: ./credstore_admin.pl add --server [FQDN] --username [ME] --password [PASSWD]
- To join domain: sudo domainjoin-cli join nazaudy.internal
This email address is being protected from spambots. You need JavaScript enabled to view it. - After it has successfully joined, run sudo reboot to restart vMA
- Ensure the servers (and obviously vCenter) are added to the domain first
- Type vifp addserver host1.nazaudy.internal --authpolicy adauth --username nazaudy\\Administrator
- Type vifp listservers to ensure the server has been added succesfully
- Type vifptarget -s host1.nazaudy.internal to connect to the server and start issuing commands
London, 25 August 2017
If you liked this article of mine about the VCP550D Delta Exam Technical lab, here there are articles regarding VMware too that you may like! :)
- VPC550D Delta Exam Technical lab https://www.nazaudy.com/vcp550d-delta-exam-technical-lab
- ESXi trunk to Cisco C2960 switch - How to load balance traffic https://www.nazaudy.com/esxi-trunk-to-cisco-c2960-switch-how-to-load-balance-traffic
Comments powered by CComment