Rethinking Network Design with FlexFabric and Vsphere 4.1

The biggest changes we have seen is the with Vsphere is the vDS (distributed virtual switch), and FT (fault tolerance).  One of the biggest changes we have with the latest version of Vsphere 4.1 on the network side of the house is NET I/O Control NetIOC, not to mention the not often mentioned LBT (load-balanced teaming).   Also while these changes occur with VMware we see changes in the HP virtual 10GB networking bringing us Flex-Fabric which is enough change to really draw some confusion…I see these changes as bringing a certain synergy to the datacenter from a HP Blade prospective with Vsphere 4.1 implementations.  I also see a serious cost savings and increased efficiency not to mention cleaner design.

Here is the look from the physical topology:

Flex-Fabric brings HPs Flex-10 Converged I/O and 1 HOP FCOE while, not as far as Cisco maybe with there current Nexus line there is still a major cost savings to the customer.  Take for instance the current Flex-10 Virtual Connect Implementation, each c7000 would need a minimum of 4 switches two for Virtual Connect SAN connections and two for Network uplinks.  Now with converged I/O the customer could buy two switches and save roughly 40k per C7000.  The two switches would both have uplinks to both SAN and Network.

What will this look like inside of virtual connect/flex-fabric?

Instead of getting four flex Nics you will get three and one Converged Adapter.

Will this work on G5′s or G6′s, what about previous flex-10 modules?

No unfortunately only G7′s, and flex-fabric modules

Any recommendations on thoughts for a network design with Vsphere 4.1 and Flex-Fabric?

VMware provides us with this best practice document: http://www.vmware.com/files/pdf/techpaper/VMW_Netioc_BestPractices.pdf  I honestly haven’t seen the latest cookbook for virtual connect.  Hopefully it addresses the distributed virtual switch now.  This was lacking last I saw.

Utilize Vsphere Network I/O control and .1q with different dv port groups for virtual machine traffic, FT, vmotion, service console to achieve a fully dynamic use of your 10gb bandwidth and only use two uplinks (the converged I/O) in an Active/Active scenario.  I really don’t see the sense of having a scenario with 1 dVs then different uplinks to different dv port groups to different virtual flexnic uplinks since you already have features in VMware to tackle I/O contention and prioritize latency sensitive traffic like shares, limits, traffic shaping.  I would avoid the use of limits and reservations were possible.  Shares will trump limits and reservation providing a better use of capacity.

Limit VMotion through Egress Traffic Shaping at the dv port group as Ingress isn’t needed with NetIOC.  This will help in a situation with multiple vmotions from many hosts.  Picture a scenario where you place multiple hosts in a cluster in maintenance mode and it is set to fully automated DRS.  Limiting the MAX vmotion will help in ensuring latency sensitive traffic is interrupted.  The below example limits vmotion to 3GB.

Network Resource Pool	Host Limit	Physical Share	Share Value
FT	Unlimited	High	100
vMotion Traffic	3GB	Normal	50
Management	Unlimited	Normal	50
VirtualMachine Traffic	Unlimited	Custom	75

So In Active/Active Flex-10 or Flex-Fabric does this mean that it will load balance automatically?

This isn’t exactly what you think….This is where LBT steps in.  To use this select Route based on physical nic load on your dvportgroup settting for teaming and failover.  LBT will only move a flow when the mean send and receive utilization of an uplink exceeds 75% of a capacity over a 30 second period.  It won’t move it more than 30 seconds.  There may be some hidden way to adjust that setting, I just don’t know it.

The actual best practices from VMware are as follows:

NetIOC Best Practices: VMware provides us with this best practice document: http://www.vmware.com/files/pdf/techpaper/VMW_Netioc_BestPractices.pdf

Flex-Fabric Best Practices:      http://h20000.www2.hp.com/bc/docs/support/SupportManual/c02499726/c02499726.pdf

Best practice 1: When using bandwidth allocation, use “shares” instead of “limits,” as the former has greater flexibility for unused capacity redistribution. Partitioning the available network bandwidth among different types of network traffic flows using limits has shortcomings. For instance, allocating 2Gbps bandwidth by using a limit for the virtual machine resource pool provides a maximum of 2Gbps bandwidth for all the virtual machine traffic even if the team is not saturated. In other words, limits impose hard limits on the amount of the bandwidth usage by a traffic flow even when there is network bandwidth available.

Best practice 2: If you are concerned about physical switch and/or physical network capacity, consider imposing limits on a given resource pool. For instance, you might want to put a limit on vMotion traffic flow to help in situations where multiple vMotion traffic flows initiated on different ESX hosts at the same time could possibly oversubscribe the physical network. By limiting the vMotion traffic bandwidth usage at the ESX host level, we can prevent the possibility of jeopardizing performance for other flows going through the same points of contention.

Best practice 3: Fault tolerance is a latency-sensitive traffic flow, so it is recommended to always set the corresponding resource- pool shares to a reasonably high relative value in the case of custom shares. However, in the case where you are using the predefined default shares value for VMware FT, leaving it set to high is recommended.

Best practice 4: We recommend that you use LBT as your vDS teaming policy while using NetIOC in order to maximize the networking capacity utilization.

NOTE: As LBT moves flows among uplinks it may occasionally cause reordering of packets at the receiver.

Best practice 5: Use the DV Port Group and Traffic Shaper features offered by the vDS to maximum effect when configuring the vDS. Configure each of the traffic flow types with a dedicated DV Port Group. Use DV Port Groups as a means to apply configuration policies to different traffic flow types, and more important, to provide additional Rx bandwidth controls through the use of Traffic Shaper. For instance, you might want to enable Traffic Shaper for the egress traffic on the DV Port Group used for vMotion. This can help in situations when multiple vMotions initiated on different vSphere hosts converge to the same destination vSphere server.