This advisory was released today  for Virtual Connect you can read about it here:

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c02720395&lang=en&cc=us&taskId=101&prodSeriesId=3540808&prodTypeId=329290

SUPPORT COMMUNICATION – CUSTOMER ADVISORY Document ID: c02720395 Version: 3 Advisory: (Revision) HP Virtual Connect – Virtual Connect Manager May Be Unable to Communicate (NO_COMM) if DNS Is Enabled for Virtual Connect Ethernet Modules NOTICE: The information in this document, including products and software versions, is current as of the Release Date. This document is subject to change without notice. Release Date: 2011-03-07 Last Updated: 2011-03-07 DESCRIPTION Document Version Release Date Details 3 03/07/2011 Added clarifications to the three scenarios described in the Resolution section to ensure the full sequence of steps is followed. 2 03/04/2011 Added additional details regarding the circumstances in which the issue may occur. Also, added three different workaround scenarios depending on whether Enclosure Bay IP Addressing (EBIPA) or external DHCP is being used and the version of OA firmware that is in use. 1 02/14/2011 Original Document Release. The HP Virtual Connect Manager (VCM) may not be able to communicate (NO_COMM) with Virtual Connect (VC) Ethernet modules in an HP BladeSystem c-Class enclosure or multiple enclosures that are part of the same Virtual Connect Domain. The loss of communication may occur in a new or an existing environment when a VCM Administrator attempts to perform any of the following tasks: Firmware Update Add/remove/reset server blades or Onboard Administrator (OA) modules Retrieve any VC Ethernet module status and state information (e.g. stacking links, port statistics, etc.) Add/edit/copy/delete/assign Server Profile Add/edit/delete VC Network Configure Port Mirroring Restore Domain Configuration Change SNMP Settings Change Advanced Ethernet Settings IMPORTANT: Attempting to execute any of the above tasks during NO_COMM adds additional risk of a network outage during the recovery steps described below. Customers particularly susceptible to this issue have VC Modules with management IP Addresses configured in the 10.x.x.x range and configured for DNS. When this problem occurs, the VC Manager will still be accessible, but all VC Ethernet modules in the domain will be displayed with an Overall Status of “No Communication.” The Virtual Connect Domain will show a “failed” status, stacking links will show “failed” and Profiles and Networks will show a status of “Unknown.” When in this state, server blades in the Virtual Connect domain will still be able to pass data traffic with no impact. This occurs if DNS is enabled for the primary VC module. The VCM may initiate a DNS reverse lookup for a very limited scope of incorrect IP addresses for the VC Ethernet modules. If this reverse lookup fails, (i.e., it is not answered by the DNS infrastructure), the primary VC module will be able to communicate correctly with the VC Ethernet modules. If the DNS infrastructure responds to this incorrect DNS reverse lookup, then VCM attempts to communicate with the VC Ethernet modules on this incorrect IP Address and fails, triggering a NO_COMM condition. Recently, the global DNS infrastructure began responding to these limited DNS reverse lookups. SCOPE Any HP Virtual Connect Ethernet Modules in a c-Class BladeSystem enclosure running VC Firmware Version 1.3x, 2.x or 3.x (up to and including 3.15) RESOLUTION A future version of Virtual Connect firmware (currently planned for release in mid-March 2011) will prevent this issue from occurring. As a workaround, disable DNS for the Virtual Connect Ethernet Modules in Enclosure Bay IP Addressing (EBIPA) or external DHCP. Removing DNS from VC Modules can potentially impact the following Virtual Connect features, if configured to use DNS names: Directory Server Settings – If a DNS name is configured for the Directory Server Address then it will no longer be resolved. The IP address will need to be configured as the Directory Server Address. SNMP Trap Destination – If a DNS name is configured for the SNMP Trap Destination then it will no longer be resolved. The IP address will need to be configured as the SNMP Trap Destination. From the VCM CLI – Any URL targets provided to save backup configuration or support dump will need to use IP address and not DNS name. The following three workaround scenarios depend on whether Enclosure Bay IP Addressing (EBIPA) or external DHCP is being used and the version of Onboard Administrator (OA) firmware that is in use: Scenario 1 – Enclosure Bay IP Addressing is being used to provide IP Addresses to the VC Ethernet Modules and the OA firmware version is 3.00 (or higher): Log into the Onboard Administrator and select Enclosure Settings > Enclosure Bay IP Addressing. Select the “Interconnect Bays” tab and remove DNS server IP address entries from the bays that include VC Ethernet Modules and click Apply. Within 5 minutes, the DNS settings for the modules should update and normal module communication will be restored. It is important that no VC domain changes are made until the following step is fully completed. From the VCM GUI, select “Tools” => “Reset Virtual Connect Manager.” This will force resynchronization of the modules if not synchronized in Step 3 above.IMPORTANT : If the NO_COMM condition was present or detected during one of the VCM administrative update tasks (listed in the DESCRIPTION section above), VCM may automatically resynchronize the modules, which would create a temporary VC domain-wide network outage during VC module initialization in either Step 3 or Step 5 above (but not both). Outage time will vary depending on the size of the VC domain. Scenario 2 - Enclosure Bay IP Addressing is being used to provide IP Addresses to the VC Ethernet Modules and the OA firmware version is 2.60 (or earlier). If iLO DNS name registrations are statically assigned in the DNS infrastructure, move to Step 3 below: If relying on Dynamic DNS updates for iLO, the OA firmware version must be updated to at least OA FW 3.11 before proceeding with the next step, otherwise iLO will only be reachable by IP address and there may be other ramifications to iLO LDAP Authentication. In the OA, in Enclosure Bay IP Addressing, Select the “Interconnect Bays” tab and remove the DNS server IP address entries from the “Shared Interconnect Settings.” Click Apply. In the OA, in Enclosure Bay IP Addressing, Select the “Device Bays” tab and remove DNS server IP address entries from the “Shared Interconnect Settings” and click Apply. Within 5 minutes, the DNS settings for the modules should update and normal module communication will be restored. It is important that no VC domain changes are made until the following step is fully completed. From the VCM GUI, select “Tools” => “Reset Virtual Connect Manager.” This will force resynchronization of the modules if not synchronized in Step 3 above.IMPORTANT : If the NO_COMM condition was present or detected during one of the VCM administrative update tasks (listed in the DESCRIPTION section above), VCM may automatically resynchronize the modules, which would create a temporary VC domain-wide network outage during VC module initialization in either Step 4 or Step 6 above (but not both). Outage time will vary depending on the size of the VC domain. Scenario 3 - External DHCP is being used to provide IP Addresses to the VC Ethernet Modules with any version of OA firmware: On the External DHCP Scope, create an exclusion range of IP addresses (preferably only the VC Ethernet module addresses). This exclusion range needs to be configured within EBIPA on the OA. In the OA, in Enclosure Bay IP Addressing, select the “Interconnect Bays” tab and configure the IP Addresses that were excluded in Step 1 above for bays that contain VC Ethernet modules. Do not configure DNS Server entries. Click Apply. It is important that no VC domain changes are made until the following steps are fully completed. Reboot the standby and primary VC modules to force them to use the new EBIPA lease. In a redundant design, the modules should be rebooted serially to mitigate downtime. a. Reset the standby VC Module from OA. b. Wait 15 minutes for the standby module to recover. c. Reset the primary VC Module from OA. d. Within 5 minutes, normal module communication will be restored. IMPORTANT : If the NO_COMM condition was present or detected during one of the VCM administrative update tasks (listed in the DESCRIPTION section above), VCM may automatically resynchronize the modules, which would create a temporary VC domain-wide network outage during VC module initialization. Outage time will vary depending on the size of the VC domain.

The biggest changes we have seen is the with Vsphere is the vDS (distributed virtual switch), and FT (fault tolerance).  One of the biggest changes we have with the latest version of Vsphere 4.1 on the network side of the house is NET I/O Control NetIOC, not to mention the not often mentioned LBT (load-balanced teaming).   Also while these changes occur with VMware we see changes in the HP virtual 10GB networking bringing us Flex-Fabric which is enough change to really draw some confusion…I see these changes as bringing a certain synergy to the datacenter from a HP Blade prospective with Vsphere 4.1 implementations.  I also see a serious cost savings and increased efficiency not to mention cleaner design.

Here is the look from the physical topology:

Flex-Fabric brings HPs Flex-10 Converged I/O and 1 HOP FCOE while, not as far as Cisco maybe with there current Nexus line there is still a major cost savings to the customer.  Take for instance the current Flex-10 Virtual Connect Implementation, each c7000 would need a minimum of 4 switches two for Virtual Connect SAN connections and two for Network uplinks.  Now with converged I/O the customer could buy two switches and save roughly 40k per C7000.  The two switches would both have uplinks to both SAN and Network.

What will this look like inside of virtual connect/flex-fabric?

Instead of getting four flex Nics you will get three and one Converged Adapter.

Will this work on G5′s or G6′s, what about previous flex-10 modules?

No unfortunately only G7′s, and flex-fabric modules

Any recommendations on thoughts for a network design with Vsphere 4.1 and Flex-Fabric?

VMware provides us with this best practice document: http://www.vmware.com/files/pdf/techpaper/VMW_Netioc_BestPractices.pdf  I honestly haven’t seen the latest cookbook for virtual connect.  Hopefully it addresses the distributed virtual switch now.  This was lacking last I saw.

Utilize Vsphere Network I/O control and .1q with different dv port groups for virtual machine traffic, FT, vmotion, service console to achieve a fully dynamic use of your 10gb bandwidth and only use two uplinks (the converged I/O) in an Active/Active scenario.  I really don’t see the sense of having a scenario with 1 dVs then different uplinks to different dv port groups to different virtual flexnic uplinks since you already have features in VMware to tackle I/O contention and prioritize latency sensitive traffic like shares, limits, traffic shaping.  I would avoid the use of limits and reservations were possible.  Shares will trump limits and reservation providing a better use of capacity.

Limit VMotion through Egress Traffic Shaping at the dv port group as Ingress isn’t needed with NetIOC.  This will help in a situation with multiple vmotions from many hosts.  Picture a scenario where you place multiple hosts in a cluster in maintenance mode and it is set to fully automated DRS.  Limiting the MAX vmotion will help in ensuring latency sensitive traffic is interrupted.  The below example limits vmotion to 3GB.

So In Active/Active Flex-10 or Flex-Fabric does this mean that it will load balance automatically?

This isn’t exactly what you think….This is where LBT steps in.  To use this select Route based on physical nic load on your dvportgroup settting for teaming and failover.  LBT will only move a flow when the mean send and receive utilization of an uplink exceeds 75% of a capacity over a 30 second period.  It won’t move it more than 30 seconds.  There may be some hidden way to adjust that setting, I just don’t know it.

The actual best practices from VMware are as follows:

NetIOC Best Practices: VMware provides us with this best practice document: http://www.vmware.com/files/pdf/techpaper/VMW_Netioc_BestPractices.pdf

Flex-Fabric Best Practices:      http://h20000.www2.hp.com/bc/docs/support/SupportManual/c02499726/c02499726.pdf

Best practice 1: When using bandwidth allocation, use “shares” instead of “limits,” as the former has greater flexibility for unused capacity redistribution. Partitioning the available network bandwidth among different types of network traffic flows using limits has shortcomings. For instance, allocating 2Gbps bandwidth by using a limit for the virtual machine resource pool provides a maximum of 2Gbps bandwidth for all the virtual machine traffic even if the team is not saturated. In other words, limits impose hard limits on the amount of the bandwidth usage by a traffic flow even when there is network bandwidth available.

Best practice 2: If you are concerned about physical switch and/or physical network capacity, consider imposing limits on a given resource pool. For instance, you might want to put a limit on vMotion traffic flow to help in situations where multiple vMotion traffic flows initiated on different ESX hosts at the same time could possibly oversubscribe the physical network. By limiting the vMotion traffic bandwidth usage at the ESX host level, we can prevent the possibility of jeopardizing performance for other flows going through the same points of contention.

Best practice 3: Fault tolerance is a latency-sensitive traffic flow, so it is recommended to always set the corresponding resource- pool shares to a reasonably high relative value in the case of custom shares. However, in the case where you are using the predefined default shares value for VMware FT, leaving it set to high is recommended.

Best practice 4: We recommend that you use LBT as your vDS teaming policy while using NetIOC in order to maximize the networking capacity utilization.

NOTE: As LBT moves flows among uplinks it may occasionally cause reordering of packets at the receiver.

Best practice 5: Use the DV Port Group and Traffic Shaper features offered by the vDS to maximum effect when configuring the vDS. Configure each of the traffic flow types with a dedicated DV Port Group. Use DV Port Groups as a means to apply configuration policies to different traffic flow types, and more important, to provide additional Rx bandwidth controls through the use of Traffic Shaper. For instance, you might want to enable Traffic Shaper for the egress traffic on the DV Port Group used for vMotion. This can help in situations when multiple vMotions initiated on different vSphere hosts converge to the same destination vSphere server.

HP released an Advisory for Flex-10 with ESX /ESXi 4.1

I still see and read on the forums that there is a lot of confusion in what the requirements are with HP Flex-10 and VMware Vsphere…..Back in January, some people were using beacon probing to be sucessful in active/active configurations however it isn’t needed today.  In order for success and to avoid a lot of confusion here is a quick summation of what you need to be successful with active/active HP Flex-10 and VMware Vsphere.

The three components for success are:

1) You need to use the VMware 1.52 driver http://downloads.vmware.com/d/details/esx_esxi_40_broadcom_bnx2_dt/ZCV0YmRqZHRidHdw

Now just to make it interesting you may run across an issue with VMware stating it isnt an authorized driver cd….You can fix that by following this:  http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1017401

2) Flex-10 should be upgraded to 2.30 or higher 3.x is out….http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareIndex.jsp?lang=en&cc=us&prodNameId=3794431&prodTypeId=3709945&prodSeriesId=3794423&swLang=13&taskId=135&swEnvOID=1005

3.) Your nic bootcode version should also be higher than 5.0.11…This is really easy even the firmware maintainence dvd from last Januarary had it….Best part is the firmware maintainance dvd is unintended install.

http://h18004.www1.hp.com/products/servers/management/core-management-100.html

Note: If you missed Part I and Part II the both outline the overall hardware topology of the 2 c7000 storage and networking both from a core router perspective to HP Flex-10 and Virtual Connect.

You can find PartI here and Part II here.

Moving on from where we left off with virtual connect and defined server profiles ESX/ESXi will need to be installed.  Once installed you will have your host with 1 nic assigned to one virtual switch for the service console without any redundant nics.

Unclaimed Network Adapters should look like this picture below.

• Vmnic1 will be assigned to the service console
• Vmnic 2 and Vmnic 3 will be used in a Dvs for Vmotion
• VMnic 4 and Vmnic 5 will be usedin a Dvs  for Fault Tolerance
• VMnic 6 and 7 will be used in a Dvs for virtual machine network traffic with port groups for both production and development virtual machines.

Step 1: Create a redundant service console.

The first step should be assigning VMnic1 to the service console available network adapters on each ESX host via the vswitch0 the regular virtual switch.  This is the same as ESX version 3.x and below.  You may also want to take out any virtual machine network that was assigned to that switch in install.

Step 2:  Create a Dvs for Vmotion

Switching views inside of virtual center to inventory networking you will then want to create a new vNetwork distributed switch.  You can customize your name to what makes sense.  Then select the appropriate amount of network uplinks, in our example that is 2 one for the e or f side.  Next you will want to assign specifically the adapters for vmotion….Now the adapters are the same as listed above (vmnic 2 and 3) for each host regardless of being in which chassis will have the ports configured as above just like our server profiles.   This makes it really easy.  Once the switch is created it would be a good idea to rename the port group to something logical for your enviroment.

Next switch views in virtual center to inventory->hosts and cluster->(host)->configuration->networking->distributed virtual switch and define your vmkernal for vmotion.  This is the same as a standard vswitch and has to be done on each host.  Again, since we didnt assign the vmotion or FT networks to a shared uplink the networks will only communicate between the flex-10 switches.  I like this for added security.  This means you can use any non-routable address scheme (e.g. 192.168.99.0-255/255.255.255.0).

Step 3:  Create a Dvs for Fault Tolerance.

This is the exact same as the Vmotion example.  Switching views inside of virtual center to inventory->networking you will then want to create a new vNetwork distributed switch.  You can customize your name to what makes sense.  Then select the appropriate amount of network uplinks, in our example that is 2 one for the e or f side.  Next you will want to assign specifically the adapters for fault tolerance….Now the adapters are the same as listed above (vmnic 4 and 5) for each host regardless of being in which chassis will have the ports configured as above just like our server profiles.   This makes it really easy.  Once the switch is created it would be a good idea to rename the port group to something logical for your enviroment.

Next switch views in virtual center to inventory->hosts and cluster->(host)->configuration->networking->distributed virtual switch and define your vmkernal for fault tolerance.  This is the same as a standard vswitch and has to be done on each host.  It is very similar to vmotion instead of checking the box for vmotion check the box for fault tolerance.  Again, since we didn’t assign the vmotion or FT networks to a shared uplink the networks will only communicate between the flex-10 switches.  I like this for added security.  This means you can use any non-routable address scheme (e.g. 192.168.98.0-255/255.255.255.0).

Step 4: Creating a Dvs for Virtual Machine traffic with 2 Vlans

This is similar to the above examples, but different in that our Dvs will have 2 port groups each specifically mapping the Vlan for the routed traffic since there connected to shared uplinks….Going with our first post lets say vlan 96 is a development network and 97 is a production network….I like to create seperate Vlans for ease of use for ACLs etc…

Switching views inside of virtual center to inventory->networking you will then want to create a new vNetwork distributed switch.  You can customize your name to what makes sense.  Then select the appropriate amount of network uplinks, in our example that is 2 one for the a or b side or c and d (depending which chassis the blade resides).  Next you will want to assign specifically the adapters for virtual machine traffic….Now the adapters are the same as listed above (vmnic 6 and 7) for each host regardless of being in which chassis will have the ports configured as above just like our server profiles.   This makes it really easy.  Once the switch is created rename the port group created to something logical like productionvms and assign the corresponding VLAN id 97 in this example.  Then click the switch and add another port group rename it to something logical like developmentvms and assign the corresponding VLAN id 96 in this example.

Now when you create a new virtual machine or move a virtual machine to the new cluster you will need to specify the network port group on the virtual machine for it to communicate on.

Note: If you missed Part 1 of this series please look here to get the topology and hardware configuration.

Step 1: The first thing to configure is your virtual connect domain.  Basically you need to follow the gui and get both your enclosures to be seen under one one virtual connect manager.  One there you can build your SAN and Ethernet configurations.  This is fairly straightforward.

Step2: Looking first at the SAN side of the configuration you will need to decide if you want to use the actual WWN or Virtual Connect supplied idea names.  I always pick the virtual connect id names, this allows for additional functionality like hardware replacements or additions dynamic not requiring a manual configuration errors, or if you plan on failing over your complete Virtual Connect enviroment pick this option.  Looking at the picture below we will be making two SAN fabrics A and B.  SAN Fabric A exists of ports 1-8 of bay 3 for Ch11 and 12 where SAN Fabric B exists of ports 1-8 of bay 4 for Ch11 and 12.

Step 3: Network Settings

Note: This sections assumes your using my previous networking configuration.  See picture below:

Similarly to the SAN you can select from either factory defined or Virtual Connect assigned MAC addresses.  Just like the SAN pick Virtual Connect assigned MAC address so that you can easily replace hardware without reconfiguring, or if you will be doing fail over.  The other settings to check is Mapped Vlans and also fast mac switching (found in the advance tab on Virtual Connect > 2.33)

After the initial networking configuration is done we need to add our shared uplinks, labeled in our original topology that would be A,B,C,D.  We will be using the Uplink SetName ESX_Network_A for Ch11 Bay1 port 1 and 2, ESX_Network_B for Ch11 Bay2 port 1 and 2, ESX_Network_C for Ch12 Bay1 port 1 and 2, ESX_Network_D for Ch12 Bay2 port 1 and 2.  The example below shows a shared uplink configuration.

After creating uplinks we will need to define the Ethernet Networks, we will be creating an Ethernet Network for each VLAN and assigning it to an uplink.  For Vmotion and FT we will be non routable networks existent only in the virtual connect domain.

To define a network name it then click the smart link check box and assign the shared uplink set.  Each Network will need to be reproduced 4 times and assigned to the corresponding uplink.  For example.  ESX_Service_Console_A will  need the corresponding shared uplink ESX_Network_A.  This should get done for each VM network as well.  When adding Vmotion and FT the same A, B,C, and D nomenclature can be used, however since these networks won’t leave the chassis they will not be assigned to any Shared Uplink Sets.

Step 4: Server Profiles

After all the networks are setup you can create your server profiles.  To do this first we need to map out how our VMware virtual switches will look and also how much bandwidth, what vlans will be used, and the speed to each switch.

This diagram shows the configuration of a HP Flex-10 blade component consisting of the 2 physical LOMs with 8 virtual network adapters or flex nics.  With these 8 flex nics 6 go to 3 Dvs, one for fault tolerance, one for vmotion, and one for virtual machines, the last two are for a standard virtual switch for the service console.  Both Fault Tolerance and Vmotion network switches are only routable between the two chassis.  Each network is set to its own bandwidth.

From Virtual Connect this is really straight forward now each switch needs to have  a corresponding network that was predefined and 1 for each side so for your top chassis you would used networks with the _A or _B, and the bottom networks would use _C and _D  for routed traffic e.g. service console and virtual machine traffic.  Use E and F for FT and Vmotion traffic that only exists with stacked Virtual Connect switches.  Please see the diagram below showing the configuration for the bottom chassis.  After the networking is set up, assignment for both the Fabric A side and B side needs to be completed.  When finished  you can assign the profile to a blade.  Keep in mind you can also copy profiles to speed up assignment….

This is an overview of a Flex-10 (active/active) stacked HP C7000′s for Vsphere 4.  There will be several posts following suit outlining the setup configuration….

Utilizing HP Virtual Connect and Flex-10 Networking enabled a very clean and efficient enviroment from the aspect of utilizing the two built in network loms both 10gb and providing 10GB uplinks to a core router instead of a cabling mess.

Many resources are available online to get the design that you want although in this example were veering from “the docs” to provide a separate non uplinked Vmotion and Fault Tolerance network for added security.  Since these networks can’t leave the chassis they are more secure.  From a VMware prospective we end up with 4 switches 1 standard vs for the service console allocated at 1gb with 4 uplink (1 to each flex-10 switch), 1 dVS for fault tolerance allocated with 1gb with 4 uplinks (1 to each flex-10 switch),1 dVS for vmotion allocated with 2gb with 4 uplinks (1 to each flex-10 switch), and 6gb to a dVS for VMware networks with multiple Vlans separated out for production and non-production networks.

Core Router Configuration:

Since port channel is Cisco exclusive LACP must be used the upstream configuration is fairly straight forward so we have the following scenario three networks 3 Vlans needed 1 for production VMs (Vlan 95), 1 for development VMs (Vlan 94), and 1 for management network (Vlan96 used for service console) again we won’t be creating uplinks from the Flex-10 switches for fault tolerance or VMotion so no configuration is needed.

Here is the config:

interface TenGigabitEthernet6/1
description HP Flex-10 NIC (left) Port 2
switchport
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 94,95,96
switchport mode trunk
channel-protocol lacp
channel-group 1 mode active
spanning-tree portfast edge

switchport
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 94.95.96
switchport mode trunk
spanning-tree portfast edge