The biggest changes we have seen is the with Vsphere is the vDS (distributed virtual switch), and FT (fault tolerance).  One of the biggest changes we have with the latest version of Vsphere 4.1 on the network side of the house is NET I/O Control NetIOC, not to mention the not often mentioned LBT (load-balanced teaming).   Also while these changes occur with VMware we see changes in the HP virtual 10GB networking bringing us Flex-Fabric which is enough change to really draw some confusion…I see these changes as bringing a certain synergy to the datacenter from a HP Blade prospective with Vsphere 4.1 implementations.  I also see a serious cost savings and increased efficiency not to mention cleaner design.

Here is the look from the physical topology:

Flex-Fabric brings HPs Flex-10 Converged I/O and 1 HOP FCOE while, not as far as Cisco maybe with there current Nexus line there is still a major cost savings to the customer.  Take for instance the current Flex-10 Virtual Connect Implementation, each c7000 would need a minimum of 4 switches two for Virtual Connect SAN connections and two for Network uplinks.  Now with converged I/O the customer could buy two switches and save roughly 40k per C7000.  The two switches would both have uplinks to both SAN and Network.

What will this look like inside of virtual connect/flex-fabric?

Instead of getting four flex Nics you will get three and one Converged Adapter.

Will this work on G5′s or G6′s, what about previous flex-10 modules?

No unfortunately only G7′s, and flex-fabric modules

Any recommendations on thoughts for a network design with Vsphere 4.1 and Flex-Fabric?

VMware provides us with this best practice document: http://www.vmware.com/files/pdf/techpaper/VMW_Netioc_BestPractices.pdf  I honestly haven’t seen the latest cookbook for virtual connect.  Hopefully it addresses the distributed virtual switch now.  This was lacking last I saw.

Utilize Vsphere Network I/O control and .1q with different dv port groups for virtual machine traffic, FT, vmotion, service console to achieve a fully dynamic use of your 10gb bandwidth and only use two uplinks (the converged I/O) in an Active/Active scenario.  I really don’t see the sense of having a scenario with 1 dVs then different uplinks to different dv port groups to different virtual flexnic uplinks since you already have features in VMware to tackle I/O contention and prioritize latency sensitive traffic like shares, limits, traffic shaping.  I would avoid the use of limits and reservations were possible.  Shares will trump limits and reservation providing a better use of capacity.

Limit VMotion through Egress Traffic Shaping at the dv port group as Ingress isn’t needed with NetIOC.  This will help in a situation with multiple vmotions from many hosts.  Picture a scenario where you place multiple hosts in a cluster in maintenance mode and it is set to fully automated DRS.  Limiting the MAX vmotion will help in ensuring latency sensitive traffic is interrupted.  The below example limits vmotion to 3GB.

So In Active/Active Flex-10 or Flex-Fabric does this mean that it will load balance automatically?

This isn’t exactly what you think….This is where LBT steps in.  To use this select Route based on physical nic load on your dvportgroup settting for teaming and failover.  LBT will only move a flow when the mean send and receive utilization of an uplink exceeds 75% of a capacity over a 30 second period.  It won’t move it more than 30 seconds.  There may be some hidden way to adjust that setting, I just don’t know it.

The actual best practices from VMware are as follows:

NetIOC Best Practices: VMware provides us with this best practice document: http://www.vmware.com/files/pdf/techpaper/VMW_Netioc_BestPractices.pdf

Flex-Fabric Best Practices:      http://h20000.www2.hp.com/bc/docs/support/SupportManual/c02499726/c02499726.pdf

Best practice 1: When using bandwidth allocation, use “shares” instead of “limits,” as the former has greater flexibility for unused capacity redistribution. Partitioning the available network bandwidth among different types of network traffic flows using limits has shortcomings. For instance, allocating 2Gbps bandwidth by using a limit for the virtual machine resource pool provides a maximum of 2Gbps bandwidth for all the virtual machine traffic even if the team is not saturated. In other words, limits impose hard limits on the amount of the bandwidth usage by a traffic flow even when there is network bandwidth available.

Best practice 2: If you are concerned about physical switch and/or physical network capacity, consider imposing limits on a given resource pool. For instance, you might want to put a limit on vMotion traffic flow to help in situations where multiple vMotion traffic flows initiated on different ESX hosts at the same time could possibly oversubscribe the physical network. By limiting the vMotion traffic bandwidth usage at the ESX host level, we can prevent the possibility of jeopardizing performance for other flows going through the same points of contention.

Best practice 3: Fault tolerance is a latency-sensitive traffic flow, so it is recommended to always set the corresponding resource- pool shares to a reasonably high relative value in the case of custom shares. However, in the case where you are using the predefined default shares value for VMware FT, leaving it set to high is recommended.

Best practice 4: We recommend that you use LBT as your vDS teaming policy while using NetIOC in order to maximize the networking capacity utilization.

NOTE: As LBT moves flows among uplinks it may occasionally cause reordering of packets at the receiver.

Best practice 5: Use the DV Port Group and Traffic Shaper features offered by the vDS to maximum effect when configuring the vDS. Configure each of the traffic flow types with a dedicated DV Port Group. Use DV Port Groups as a means to apply configuration policies to different traffic flow types, and more important, to provide additional Rx bandwidth controls through the use of Traffic Shaper. For instance, you might want to enable Traffic Shaper for the egress traffic on the DV Port Group used for vMotion. This can help in situations when multiple vMotions initiated on different vSphere hosts converge to the same destination vSphere server.

I recently came across a few issues implementing Netapp Virtualization Storage Console with Vsphere 4.1….When adding filers to the provisioning and cloning  section when timeout occur or you get the generic message “A general Error has occured” try the following work around steps:

1) Does your filer have NFS enabled even if your not using enable it support can provide you a temporary key

https://now.netapp.com/Knowledgebase/solutionarea.asp?id=kb59430

2) e0M interface issue should be temporarily disabled

: bug # 320355

http://now.corp.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=320355

3) Increasing ZAPI timeout via VSCPreferences file is in the KB below, and also in VSC’s known issues, and IAG on the NOW site. Links below:

https://now.netapp.com/Knowledgebase/solutionarea.asp?id=kb59154

http://now.netapp.com/NOW/download/software/vsc_win/2.0/

4) Enable SSL 2.0 in webbrowser

I was troubleshooting an issue with hardware compatability with Fault Tolerance when I ran into this neat utility (site survery)  A link to it can be found here on the Vmware site.  Once installed when you close your Vcenter client and open it back up, click on a cluster and then the site survey tab.  A report will generate after a few minutes….

Also on this site is a cpu identification utility for Vmotion compatability testing etc

Here is a quick cheat sheet for the vMA (if you need it).

• The default user name is vi-admin password is set on install
• To add an esx host type vifp addserver <your esxhostname.company.com>
• To show a quick list of your servers vifp listservers
• To initialize a connection to a particular host type vifpinit esxhostname.company.com
• To capture logs from an esx host type vilogger enable –server <fqdn of esxhost you want to monitor> –numrotation 30 –maxfilesize 1023 –collectionperiod 100

How can I launch resxtop remotely?

$resxtop –server <fqdn of your vCenter> –vihost <fqdn of esxhost you want to monitor> –username <your username to login Virtual Center>

(Note: you will be prompted for your password)

How can I run resxtop in batch mode and store all that in a .csv ?  The command below batches it takes 60 samples every 5 seconds then stores it in a file named data.csv

$resxtop -b -a -n 60 –server <fqdn of your virtualcenter server> –vihost <fqdn of your esxhost> –username (for your virtualcenter)

Why would I ever want to run resxtops when this stuff is in vCenter?  lol.

Some quick notes now that were in esxtops remotely using the vMA….

1) resxtops updates every 5 seconds to delay it type s then the refresh interval (20 would be 20 seconds)
2) Type V to just show virtual machines
3) To drill down into a virtual machine and look at the worlds type e then the gid
4) typing c will bring up cpu, m will bring up memory, d will bring up disk and n will bring up network

Troubleshooting Memory from the vMA with resxtops

1) Determine if the balloon driver is installed in a Virtual Machine  type m for memory view then f to toggle fields select MCTL
–Now that it is selected return back to the screen looking at the MCTL if the MCTL collumn says N on a virtual machine then the balloon driver isnt installed

2) Look at MEMSZ and GRANT counters…GRANT\MEMSZ = %memory used

3) To check demand of virtual machines memory  a  quick peak at Memory Usage Counter, the Average column and the Maximum (peak) column will help greatly if average> 80 or peak >90 high demand for virtual machines memory might be causing problem. ( I know virtual center)

4) Too check to see if you ESX host swapped in the past look at SWAP/MB if the value > 0 it has swapped virtual machine memory in the past.  If the answer is no, the ESX host doesnt have any virtual machine memory swapped.

5) Look at your SWCUR for your virtual machine if  the value> 0 then the ESX host has swapped memory from your test VM.

6) Look at your MCTLSZ if this > 0 your vm is balloning, if SWR/s or SWW/s your virtual machine is swapping

7) Look at MCTLSZ for your test virtual machine if value > 0 then vm is balloning

Here is a quick cheat sheet for the vMA (if you need it).

• The default user name is vi-admin password is set on install
• To add an esx host type vifp addserver <your esxhostname.company.com>
• To show a quick list of your servers vifp listservers
• To initialize a connection to a particular host type vifpinit esxhostname.company.com
• To capture logs from an esx host type vilogger enable –server <fqdn of esxhost you want to monitor> –numrotation 30 –maxfilesize 1023 –collectionperiod 100

How can I launch resxtop remotely?

$resxtop –server <fqdn of your vCenter> –vihost <fqdn of esxhost you want to monitor> –username <your username to login Virtual Center>

(Note: you will be prompted for your password)

How can I run resxtop in batch mode and store all that in a .csv ?  The command below batches it takes 60 samples every 5 seconds then stores it in a file named data.csv

$resxtop -b -a -n 60 –server <fqdn of your virtualcenter server> –vihost <fqdn of your esxhost> –username (for your virtualcenter)

Why would I ever want to run resxtops when this stuff is in vCenter?  lol.

Some quick notes now that were in esxtops remotely using the vMA….

1) resxtops updates every 5 seconds to delay it type s then the refresh interval (20 would be 20 seconds)
2) Type V to just show virtual machines
3) To drill down into a virtual machine and look at the worlds type e then the gid
4) typing c will bring up cpu, m will bring up memory, d will bring up disk and n will bring up network

Troubleshooting CPU Problems on ESX Hosts and VMs

The first thing you should do is look at your physical ESX host CPU

Again, Typing c will bring the CPU section of resxtops you can add and remove and rearrange fields typing f
Look at your PCPU UTIL -% if the average is over 75% or peaks >90% it is very possible ESX host CPU is saturated; service console runs at PCPU 0

The 2nd thing you should do is check Cpu Ready of your VMs if your CPU ready is for most vms is hovering around 20% or higher they are waiting too long for CPU either adjust resource pool or vmotion to a different host.

The third thing I would do it look at individual VMs CPU %USED and %RDY.  A high ready could be because the host or a resource pool make adjustments accordingly, a high %used like 75% or higher might be a sign to give it another cpu.  However is your application SMP friendly?

Does this VM really need 2 CPUs?

If it does it should be using them….right?

To check if the VM is really making use of the cpus you gave it expand a world of a VM.  Look at your vCPU( again from the resxtops cpu view type e then the gid of your vm)….if one of the vCPU is very low and one a lot higher chances are your virtual machine isn’t symetric multiprocessing and you should remove a cpu.  Giving CPUs to a VirtualMachine that isnt Symetric Multiprocessing is bad for performance….

What about swapping?
To check swapping look at the %SWPWT column above 5% performance of the VM will degrade significantly

This post assumes you already know how to configure ESX/ESXi 4.1 for Active Directory if not this will get you up and running: http://ict-freak.nl/2010/09/12/how-to-configure-vsphere-4-1-active-directory-authentication/

3 Gotchas

1) After joining ESX/ESXi hosts to the domain and listing the group or user Administrator access login failure occurs…

-Looking in the /var/log directory output is seen referencing “ESX Admins” group during the authentication failure.

Oct  1 09:27:36 hostname lsassd[13781]: 0xf7544b90:Failed to find user or group. [Error code: 40071]
Oct  1 09:28:04 hostname nssquery: Group lookup failed for ‘YourDomain\ESX Admins’

Oct  1 09:29:04 hostname nssquery: Group lookup failed for ‘YourDomain\ESX Admins’
Oct  1 09:30:05 hostname nssquery: Group lookup failed for ‘YourDomain\ESX Admins’
Oct  1 09:32:06 hostname last message repeated 2 times
Oct  1 09:34:07 hostname last message repeated 2 times
Oct  1 09:36:08hostname last message repeated 2 times

-After creating an ESX Admins group in Active Directory then assigning this to virtual center with the Administrator right authentication worked properly.

2) If you login to an ESX/ESXi 4.1 host that is authenticated and your Ad account is a member of more than 32 security groups you will either reboot or cause on non-responsive host.  VMware knowledge base article: ESX host reboots, becomes unresponsive, or experiences a purple diagnostic screen when logging into the service console

3) After “properly” authentication with Ad credentials I noticed an issue with being stuck in a home directory of / rather than /home/%username%

-Looking into this further I found a knowledge base article on

Home directories are not automatically created for Domain Users on ESX/ESXi 4.1 hosts that are joined to an Active Directory Domain

The create-homedir codepath has been disabled on ESX/ESXi 4.1.  Attempting to configure this behavior using the/etc/likewise/lsassd.conf file will not succeed.  To configure home directories for Active Directory user accounts, the directories must be manually created.

The /etc/likewise/lsassd.conf file can be modified to detail the location of the home directories once they exist by Adding or modifying these lines:

homedir-prefix = /home
homedir-template = %H/%U

This causes the homedir-prefix = /home to set the starting point for all home directories to be /home andhomedir-template = %H/%U sets the home directory to be the homedir-prefix %H followed by the user account name %U. The variable %D can also be used to substitute the Active Directory domain name into the user’s home directory.

Run these commands in sequence to restart the lsassd daemon and clear the Active Directory cache for these settings to take effect.

1. /etc/init.d/lsassd stop
2. rm /etc/likewise/db/lsass-adcache.filedb
3. /etc/init.d/lsassd start

This is the design that I constructed and implemented for my last companies Vsphere 4.0 Update 2 upgrade and hardware refresh for production virtual environment, I created two highly available vSphere clusters which I like to call “vClusters” using the latest HP blade technology with HP Virtual Connect and Flex-10. I was able to create a very dynamic system with 2 clusters which could easily be scaled to 4.

Hardware:

• 2 HP Blade Chassis each equipped with 2 Flex-10 and 2 8gb Virtual Connect
• Each Chassis is interconnected with 4 CX-4 stacking cables 2 per per Chassis side running between the Flex-10 modules
• 18 Bl 460s G6 each with Intel Westmere Nehalems 32 nms 6 core procs each equipped with 48gb of memory
• SAN 2 HP EVA 8400s
• SAN Core Brocade 48000 (4GB director series)
• Networking Core Cisco 6509s
• 1 DataDomain DD 560

VMware Environment:

• Licensing – All Enterprise Plus for dvs, host profiles, storage i/o (future), 12 core processors (future)
• Each Cluster will hold 100-125 Virtual Machines with room for more than double the capacity
• VMware thin provisioning (reduced storage by more than 200%)
• Estimated capacity max per blade 30 VMS
• 2 vClusters each with 8 servers 1 Server for HA reserved; fully automated DRS with DPM configured (not fully automated)
• 2 Sandbox Servers Clustered with Private Virtual Honeypot
• VMs each upgrades to virtual hardware 7 with VMware vmxnet 3
• Vranger Pro 4.5
• 4 resource pools per cluster

1. Templates – CPU and Share Resources kept to a minimum. The templates are actually powered on VM’s why? Who likes patching
2. Delete – A resource pool with no resources mainly used to put VMs that are powered off and waiting to be deleted
3. Prod – A resource pool with shares set to high for both CPU and Memory with expandable reservation
4. Dev/Test – A resource pool with shares set to normal for both CPU and Memory with expandable reservations

Networking:

• 80 gb uplinks to core router (Cisco 6509) 20 gb trunk per flex-10 module (2 flex-10) modules per chassis.
• Flex-10 (Active/Active) 20 GB of networking to each blade with 20gb of networking between blades inter chassis (read about the configuration for Flex-10 and Virtual connect here)
• dVs Fault Tolerance -Private Network – Non Routable only communicates within Blade Chassis
• dVs Vmotion – Private Network – Non Routable only communicates within Blade Chassis
• dVs Virtual Machines- Different Port groups each for different Vlans for Dev/Test/Prod
• vS Service Console
  Note: In 4.1 I would change this design and route VMotion, and do mapped VLANS and 1 dVs for Vmotion/Service Console/Virtual Machines Dev/Virtual Machines Test…Id keep fault tolerance on a seperate private switch. However with the main dVs switch I would encorporate Network I/O control to effectively and dynamically utilize the 10gb pipe this would also solve the issue of the egress problem with flex-10 only controlling traffic one way.

Storage and Backup:

• vRanger Pro 4.5 – Installed on VMs, configured to backup vClusters 50 VMs per hour very effective 50 vms per hour backup 100% success rate on backups 0 errors or troubleshooting. I honestly never thought that I would see the day after troubleshooting VCB for 2 years backups this good.
• DD 560 set up with CIFS share for VMware backups, ESX boxes backup directly to DD560. Pre thin provision 40:1 compression ratio.
• LUNS presented to each cluster with standard size of 500gb. sVMotion capability between clusters

Note: If you missed Part 1 of this series please look here to get the topology and hardware configuration.

Step 1: The first thing to configure is your virtual connect domain.  Basically you need to follow the gui and get both your enclosures to be seen under one one virtual connect manager.  One there you can build your SAN and Ethernet configurations.  This is fairly straightforward.

Step2: Looking first at the SAN side of the configuration you will need to decide if you want to use the actual WWN or Virtual Connect supplied idea names.  I always pick the virtual connect id names, this allows for additional functionality like hardware replacements or additions dynamic not requiring a manual configuration errors, or if you plan on failing over your complete Virtual Connect enviroment pick this option.  Looking at the picture below we will be making two SAN fabrics A and B.  SAN Fabric A exists of ports 1-8 of bay 3 for Ch11 and 12 where SAN Fabric B exists of ports 1-8 of bay 4 for Ch11 and 12.

Step 3: Network Settings

Note: This sections assumes your using my previous networking configuration.  See picture below:

Similarly to the SAN you can select from either factory defined or Virtual Connect assigned MAC addresses.  Just like the SAN pick Virtual Connect assigned MAC address so that you can easily replace hardware without reconfiguring, or if you will be doing fail over.  The other settings to check is Mapped Vlans and also fast mac switching (found in the advance tab on Virtual Connect > 2.33)

After the initial networking configuration is done we need to add our shared uplinks, labeled in our original topology that would be A,B,C,D.  We will be using the Uplink SetName ESX_Network_A for Ch11 Bay1 port 1 and 2, ESX_Network_B for Ch11 Bay2 port 1 and 2, ESX_Network_C for Ch12 Bay1 port 1 and 2, ESX_Network_D for Ch12 Bay2 port 1 and 2.  The example below shows a shared uplink configuration.

After creating uplinks we will need to define the Ethernet Networks, we will be creating an Ethernet Network for each VLAN and assigning it to an uplink.  For Vmotion and FT we will be non routable networks existent only in the virtual connect domain.

To define a network name it then click the smart link check box and assign the shared uplink set.  Each Network will need to be reproduced 4 times and assigned to the corresponding uplink.  For example.  ESX_Service_Console_A will  need the corresponding shared uplink ESX_Network_A.  This should get done for each VM network as well.  When adding Vmotion and FT the same A, B,C, and D nomenclature can be used, however since these networks won’t leave the chassis they will not be assigned to any Shared Uplink Sets.

Step 4: Server Profiles

After all the networks are setup you can create your server profiles.  To do this first we need to map out how our VMware virtual switches will look and also how much bandwidth, what vlans will be used, and the speed to each switch.

This diagram shows the configuration of a HP Flex-10 blade component consisting of the 2 physical LOMs with 8 virtual network adapters or flex nics.  With these 8 flex nics 6 go to 3 Dvs, one for fault tolerance, one for vmotion, and one for virtual machines, the last two are for a standard virtual switch for the service console.  Both Fault Tolerance and Vmotion network switches are only routable between the two chassis.  Each network is set to its own bandwidth.

From Virtual Connect this is really straight forward now each switch needs to have  a corresponding network that was predefined and 1 for each side so for your top chassis you would used networks with the _A or _B, and the bottom networks would use _C and _D  for routed traffic e.g. service console and virtual machine traffic.  Use E and F for FT and Vmotion traffic that only exists with stacked Virtual Connect switches.  Please see the diagram below showing the configuration for the bottom chassis.  After the networking is set up, assignment for both the Fabric A side and B side needs to be completed.  When finished  you can assign the profile to a blade.  Keep in mind you can also copy profiles to speed up assignment….