Recently working with an ESX 4.1 upgrade I was tasked with getting System Insight Manager Agents for the appropriate HP Server installed and configured.  I found this HP Website extremly beneficial in cross referencing your HP Server Models with the supported ESX version and compatibility matrix for appropriate SIM agent version with a download link

The page is called “VMware from HP” here is the link it contains both supportability for ESX/ESXi but also FT.  This also makes it very easy to find your appropriate insight manager agents.  Worth a bookmark at the least.

BL series	VMware ESX/ESXi Server
	FT	ESXi 4.0	ESXi 4.0 U1	ESXi 4.0 U2	ESXi 4.1	ESX 4.0	ESX 4.0 U1	ESX 4.0 U2	ESX 4.1
BL2x220c G6(2)
BL260c G5
BL280c G6(2)
BL460c
BL460c G5
BL460c G6
BL465c
BL465c G5
BL465c G6
BL465c G7
BL480c
BL490c G6
BL495c G5
BL495c G6
BL680c G5
BL685c
BL685c G5
BL685c G6
BL685c G7

DL series	VMware ESX/ESXi Server
	FT	ESXi 4.0	ESXi 4.0 U1	ESXi 4.0 U2	ESXi 4.1	ESX 4.0	ESX 4.0 U1	ESX 4.0 U2	ESX 4.1
DL160 G6(1)
DL160se G6
DL165 G7
DL170h G6(1)
DL180 G6(1)
DL320 G6(1)
DL360 G5
DL360 G6
DL360 G7			(3)	(3)	(3)
DL365
DL365 G5
DL370 G6
DL380 G5
DL380 G6
DL380 G7
DL385 G2
DL385 G5p
DL385 G5
DL385 G6
DL385 G7
DL580 G4(1)
DL580 G5
DL580 G7
DL585 G2
DL585 G5
DL585 G6
DL585 G7
DL785 G5
DL785 G6
DL980 G7

ML series	VMware ESX/ESXi Server
	FT	ESXi 4.0	ESXi 4.0 U1	ESXi 4.0 U2	ESXi 4.1	ESX 4.0	ESX 4.0 U1	ESX 4.0 U2	ESX 4.1
ML150 G6(1)
ML330 G6
ML350 G5
ML350 G6
ML370 G5
ML370 G6

SL series	VMware ESX/ESXi Server
	FT	ESXi 4.0	ESXi 4.0 U1	ESXi 4.0 U2	ESXi 4.1	ESX 4.0	ESX 4.0 U1	ESX 4.0 U2	ESX 4.1
SL165z G7
SL160z G6(1)
SL170z G6(1)
SL2x170z G6

Agents	VMware ESX Server
	4.0	4.0 U1	4.0 U2	4.1
8.2.5
8.3.0
8.3.1
8.4.0
8.5.1
8.6.0

*

“Supported” indicates that an operating system has been successfully tested on the server and drivers are available. Servers are supported with one socket populated.

FT	= Fault Tolerance.

(1)	Not supported for ESXi

(2)	Requires the HP custom ESXi image available at hp.com
(3)	Need to use April 2010 image – ESXi HD-USB-SD Image Installer CD(583772-003.iso)

This post assumes you already know how to configure ESX/ESXi 4.1 for Active Directory if not this will get you up and running: http://ict-freak.nl/2010/09/12/how-to-configure-vsphere-4-1-active-directory-authentication/

3 Gotchas

1) After joining ESX/ESXi hosts to the domain and listing the group or user Administrator access login failure occurs…

-Looking in the /var/log directory output is seen referencing “ESX Admins” group during the authentication failure.

Oct  1 09:27:36 hostname lsassd[13781]: 0xf7544b90:Failed to find user or group. [Error code: 40071]
Oct  1 09:28:04 hostname nssquery: Group lookup failed for ‘YourDomain\ESX Admins’

Oct  1 09:29:04 hostname nssquery: Group lookup failed for ‘YourDomain\ESX Admins’
Oct  1 09:30:05 hostname nssquery: Group lookup failed for ‘YourDomain\ESX Admins’
Oct  1 09:32:06 hostname last message repeated 2 times
Oct  1 09:34:07 hostname last message repeated 2 times
Oct  1 09:36:08hostname last message repeated 2 times

-After creating an ESX Admins group in Active Directory then assigning this to virtual center with the Administrator right authentication worked properly.

2) If you login to an ESX/ESXi 4.1 host that is authenticated and your Ad account is a member of more than 32 security groups you will either reboot or cause on non-responsive host.  VMware knowledge base article: ESX host reboots, becomes unresponsive, or experiences a purple diagnostic screen when logging into the service console

3) After “properly” authentication with Ad credentials I noticed an issue with being stuck in a home directory of / rather than /home/%username%

-Looking into this further I found a knowledge base article on

Home directories are not automatically created for Domain Users on ESX/ESXi 4.1 hosts that are joined to an Active Directory Domain

The create-homedir codepath has been disabled on ESX/ESXi 4.1.  Attempting to configure this behavior using the/etc/likewise/lsassd.conf file will not succeed.  To configure home directories for Active Directory user accounts, the directories must be manually created.

The /etc/likewise/lsassd.conf file can be modified to detail the location of the home directories once they exist by Adding or modifying these lines:

homedir-prefix = /home
homedir-template = %H/%U

This causes the homedir-prefix = /home to set the starting point for all home directories to be /home andhomedir-template = %H/%U sets the home directory to be the homedir-prefix %H followed by the user account name %U. The variable %D can also be used to substitute the Active Directory domain name into the user’s home directory.

Run these commands in sequence to restart the lsassd daemon and clear the Active Directory cache for these settings to take effect.

1. /etc/init.d/lsassd stop
2. rm /etc/likewise/db/lsass-adcache.filedb
3. /etc/init.d/lsassd start

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1021609

What is NMI?

NMI stands for Non-Maskable Interrupt.  It can be generated by a hard memory error, or by a button in ILO.  When a hard memory error occurs you want the system to crash so that data isn’t corrupted.  When a system is in a hard hung state, and totally unresponsive, you can use the NMI facility in ILO to crash the system so that a dump is created.

Where to get the patch?

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=3709945&prodSeriesId=3884113&swItem=MTX-c2dd60fc9642432dae086527f4&prodNameId=3884114&swEnvOID=4091&swLang=13&taskId=135&mode=3

This is the design that I constructed and implemented for my last companies Vsphere 4.0 Update 2 upgrade and hardware refresh for production virtual environment, I created two highly available vSphere clusters which I like to call “vClusters” using the latest HP blade technology with HP Virtual Connect and Flex-10. I was able to create a very dynamic system with 2 clusters which could easily be scaled to 4.

Hardware:

• 2 HP Blade Chassis each equipped with 2 Flex-10 and 2 8gb Virtual Connect
• Each Chassis is interconnected with 4 CX-4 stacking cables 2 per per Chassis side running between the Flex-10 modules
• 18 Bl 460s G6 each with Intel Westmere Nehalems 32 nms 6 core procs each equipped with 48gb of memory
• SAN 2 HP EVA 8400s
• SAN Core Brocade 48000 (4GB director series)
• Networking Core Cisco 6509s
• 1 DataDomain DD 560

VMware Environment:

• Licensing – All Enterprise Plus for dvs, host profiles, storage i/o (future), 12 core processors (future)
• Each Cluster will hold 100-125 Virtual Machines with room for more than double the capacity
• VMware thin provisioning (reduced storage by more than 200%)
• Estimated capacity max per blade 30 VMS
• 2 vClusters each with 8 servers 1 Server for HA reserved; fully automated DRS with DPM configured (not fully automated)
• 2 Sandbox Servers Clustered with Private Virtual Honeypot
• VMs each upgrades to virtual hardware 7 with VMware vmxnet 3
• Vranger Pro 4.5
• 4 resource pools per cluster

1. Templates – CPU and Share Resources kept to a minimum. The templates are actually powered on VM’s why? Who likes patching
2. Delete – A resource pool with no resources mainly used to put VMs that are powered off and waiting to be deleted
3. Prod – A resource pool with shares set to high for both CPU and Memory with expandable reservation
4. Dev/Test – A resource pool with shares set to normal for both CPU and Memory with expandable reservations

Networking:

• 80 gb uplinks to core router (Cisco 6509) 20 gb trunk per flex-10 module (2 flex-10) modules per chassis.
• Flex-10 (Active/Active) 20 GB of networking to each blade with 20gb of networking between blades inter chassis (read about the configuration for Flex-10 and Virtual connect here)
• dVs Fault Tolerance -Private Network – Non Routable only communicates within Blade Chassis
• dVs Vmotion – Private Network – Non Routable only communicates within Blade Chassis
• dVs Virtual Machines- Different Port groups each for different Vlans for Dev/Test/Prod
• vS Service Console
  Note: In 4.1 I would change this design and route VMotion, and do mapped VLANS and 1 dVs for Vmotion/Service Console/Virtual Machines Dev/Virtual Machines Test…Id keep fault tolerance on a seperate private switch. However with the main dVs switch I would encorporate Network I/O control to effectively and dynamically utilize the 10gb pipe this would also solve the issue of the egress problem with flex-10 only controlling traffic one way.

Storage and Backup:

• vRanger Pro 4.5 – Installed on VMs, configured to backup vClusters 50 VMs per hour very effective 50 vms per hour backup 100% success rate on backups 0 errors or troubleshooting. I honestly never thought that I would see the day after troubleshooting VCB for 2 years backups this good.
• DD 560 set up with CIFS share for VMware backups, ESX boxes backup directly to DD560. Pre thin provision 40:1 compression ratio.
• LUNS presented to each cluster with standard size of 500gb. sVMotion capability between clusters

HP released an Advisory for Flex-10 with ESX /ESXi 4.1

In the process of completing a Vsphere 4 migration I ran into a problem where I was unable to poweron migrated machines due to file lock issues where the VM was still registered on the host in 3.5 cluster.  Through research I came across articles where customers had issues with DVR owning a lock and having similar issues. 

Here are a few tricks if you end up in file lock issues:

1. SSH into your host that your VM won’t power on. 
2. CD to your VMs directory on your VMFS datastore
3. vmkfstools -D {yourvmname}.vmdk | cat/var/log/vmkernel | grep -i owner
   Your output will return something similar to this:

Aug 13 10:12:22 ESXHOSTNAME vmkernel: gen 717, mode 1, owner
c24d759-19fb4dc3-e694-001f295d02f6 mtime 3915152674347]

    4.  Now looking at the highlighted portion that is the host MAC address that has the lock moving forward we need to check our hosts for that mac address, rather than searching the whole address the last four digits will do fine.
    5.  Now on each host that is a lock contender you will need to putty in and  run this command: ifconfig | grep -i 02:f6 two things will happen if it isnt the host you will get a command prompt; if it is the host your output will look like this:
vmnic6    Link encap:Ethernet  HWaddr 00:1F:29:5D:02:F6
   6.  Going back in virtual center select the VM and remove from inventory.
   7.  Read the virtual machine on the host that has the lock by browsing the vmfs datastore and clicking import virtual machine.
   8.  Power on your VM…

I still see and read on the forums that there is a lot of confusion in what the requirements are with HP Flex-10 and VMware Vsphere…..Back in January, some people were using beacon probing to be sucessful in active/active configurations however it isn’t needed today.  In order for success and to avoid a lot of confusion here is a quick summation of what you need to be successful with active/active HP Flex-10 and VMware Vsphere.

The three components for success are:

1) You need to use the VMware 1.52 driver http://downloads.vmware.com/d/details/esx_esxi_40_broadcom_bnx2_dt/ZCV0YmRqZHRidHdw

Now just to make it interesting you may run across an issue with VMware stating it isnt an authorized driver cd….You can fix that by following this:  http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1017401

2) Flex-10 should be upgraded to 2.30 or higher 3.x is out….http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareIndex.jsp?lang=en&cc=us&prodNameId=3794431&prodTypeId=3709945&prodSeriesId=3794423&swLang=13&taskId=135&swEnvOID=1005

3.) Your nic bootcode version should also be higher than 5.0.11…This is really easy even the firmware maintainence dvd from last Januarary had it….Best part is the firmware maintainance dvd is unintended install.

http://h18004.www1.hp.com/products/servers/management/core-management-100.html

Note: If you missed Part I and Part II the both outline the overall hardware topology of the 2 c7000 storage and networking both from a core router perspective to HP Flex-10 and Virtual Connect.

You can find PartI here and Part II here.

Moving on from where we left off with virtual connect and defined server profiles ESX/ESXi will need to be installed.  Once installed you will have your host with 1 nic assigned to one virtual switch for the service console without any redundant nics.

Unclaimed Network Adapters should look like this picture below.

• Vmnic1 will be assigned to the service console
• Vmnic 2 and Vmnic 3 will be used in a Dvs for Vmotion
• VMnic 4 and Vmnic 5 will be usedin a Dvs  for Fault Tolerance
• VMnic 6 and 7 will be used in a Dvs for virtual machine network traffic with port groups for both production and development virtual machines.

Step 1: Create a redundant service console.

The first step should be assigning VMnic1 to the service console available network adapters on each ESX host via the vswitch0 the regular virtual switch.  This is the same as ESX version 3.x and below.  You may also want to take out any virtual machine network that was assigned to that switch in install.

Step 2:  Create a Dvs for Vmotion

Switching views inside of virtual center to inventory networking you will then want to create a new vNetwork distributed switch.  You can customize your name to what makes sense.  Then select the appropriate amount of network uplinks, in our example that is 2 one for the e or f side.  Next you will want to assign specifically the adapters for vmotion….Now the adapters are the same as listed above (vmnic 2 and 3) for each host regardless of being in which chassis will have the ports configured as above just like our server profiles.   This makes it really easy.  Once the switch is created it would be a good idea to rename the port group to something logical for your enviroment.

Next switch views in virtual center to inventory->hosts and cluster->(host)->configuration->networking->distributed virtual switch and define your vmkernal for vmotion.  This is the same as a standard vswitch and has to be done on each host.  Again, since we didnt assign the vmotion or FT networks to a shared uplink the networks will only communicate between the flex-10 switches.  I like this for added security.  This means you can use any non-routable address scheme (e.g. 192.168.99.0-255/255.255.255.0).

Step 3:  Create a Dvs for Fault Tolerance.

This is the exact same as the Vmotion example.  Switching views inside of virtual center to inventory->networking you will then want to create a new vNetwork distributed switch.  You can customize your name to what makes sense.  Then select the appropriate amount of network uplinks, in our example that is 2 one for the e or f side.  Next you will want to assign specifically the adapters for fault tolerance….Now the adapters are the same as listed above (vmnic 4 and 5) for each host regardless of being in which chassis will have the ports configured as above just like our server profiles.   This makes it really easy.  Once the switch is created it would be a good idea to rename the port group to something logical for your enviroment.

Next switch views in virtual center to inventory->hosts and cluster->(host)->configuration->networking->distributed virtual switch and define your vmkernal for fault tolerance.  This is the same as a standard vswitch and has to be done on each host.  It is very similar to vmotion instead of checking the box for vmotion check the box for fault tolerance.  Again, since we didn’t assign the vmotion or FT networks to a shared uplink the networks will only communicate between the flex-10 switches.  I like this for added security.  This means you can use any non-routable address scheme (e.g. 192.168.98.0-255/255.255.255.0).

Step 4: Creating a Dvs for Virtual Machine traffic with 2 Vlans

This is similar to the above examples, but different in that our Dvs will have 2 port groups each specifically mapping the Vlan for the routed traffic since there connected to shared uplinks….Going with our first post lets say vlan 96 is a development network and 97 is a production network….I like to create seperate Vlans for ease of use for ACLs etc…

Switching views inside of virtual center to inventory->networking you will then want to create a new vNetwork distributed switch.  You can customize your name to what makes sense.  Then select the appropriate amount of network uplinks, in our example that is 2 one for the a or b side or c and d (depending which chassis the blade resides).  Next you will want to assign specifically the adapters for virtual machine traffic….Now the adapters are the same as listed above (vmnic 6 and 7) for each host regardless of being in which chassis will have the ports configured as above just like our server profiles.   This makes it really easy.  Once the switch is created rename the port group created to something logical like productionvms and assign the corresponding VLAN id 97 in this example.  Then click the switch and add another port group rename it to something logical like developmentvms and assign the corresponding VLAN id 96 in this example.

Now when you create a new virtual machine or move a virtual machine to the new cluster you will need to specify the network port group on the virtual machine for it to communicate on.

Note: If you missed Part 1 of this series please look here to get the topology and hardware configuration.

Step 1: The first thing to configure is your virtual connect domain.  Basically you need to follow the gui and get both your enclosures to be seen under one one virtual connect manager.  One there you can build your SAN and Ethernet configurations.  This is fairly straightforward.

Step2: Looking first at the SAN side of the configuration you will need to decide if you want to use the actual WWN or Virtual Connect supplied idea names.  I always pick the virtual connect id names, this allows for additional functionality like hardware replacements or additions dynamic not requiring a manual configuration errors, or if you plan on failing over your complete Virtual Connect enviroment pick this option.  Looking at the picture below we will be making two SAN fabrics A and B.  SAN Fabric A exists of ports 1-8 of bay 3 for Ch11 and 12 where SAN Fabric B exists of ports 1-8 of bay 4 for Ch11 and 12.

Step 3: Network Settings

Note: This sections assumes your using my previous networking configuration.  See picture below:

Similarly to the SAN you can select from either factory defined or Virtual Connect assigned MAC addresses.  Just like the SAN pick Virtual Connect assigned MAC address so that you can easily replace hardware without reconfiguring, or if you will be doing fail over.  The other settings to check is Mapped Vlans and also fast mac switching (found in the advance tab on Virtual Connect > 2.33)

After the initial networking configuration is done we need to add our shared uplinks, labeled in our original topology that would be A,B,C,D.  We will be using the Uplink SetName ESX_Network_A for Ch11 Bay1 port 1 and 2, ESX_Network_B for Ch11 Bay2 port 1 and 2, ESX_Network_C for Ch12 Bay1 port 1 and 2, ESX_Network_D for Ch12 Bay2 port 1 and 2.  The example below shows a shared uplink configuration.

After creating uplinks we will need to define the Ethernet Networks, we will be creating an Ethernet Network for each VLAN and assigning it to an uplink.  For Vmotion and FT we will be non routable networks existent only in the virtual connect domain.

To define a network name it then click the smart link check box and assign the shared uplink set.  Each Network will need to be reproduced 4 times and assigned to the corresponding uplink.  For example.  ESX_Service_Console_A will  need the corresponding shared uplink ESX_Network_A.  This should get done for each VM network as well.  When adding Vmotion and FT the same A, B,C, and D nomenclature can be used, however since these networks won’t leave the chassis they will not be assigned to any Shared Uplink Sets.

Step 4: Server Profiles

After all the networks are setup you can create your server profiles.  To do this first we need to map out how our VMware virtual switches will look and also how much bandwidth, what vlans will be used, and the speed to each switch.

This diagram shows the configuration of a HP Flex-10 blade component consisting of the 2 physical LOMs with 8 virtual network adapters or flex nics.  With these 8 flex nics 6 go to 3 Dvs, one for fault tolerance, one for vmotion, and one for virtual machines, the last two are for a standard virtual switch for the service console.  Both Fault Tolerance and Vmotion network switches are only routable between the two chassis.  Each network is set to its own bandwidth.

From Virtual Connect this is really straight forward now each switch needs to have  a corresponding network that was predefined and 1 for each side so for your top chassis you would used networks with the _A or _B, and the bottom networks would use _C and _D  for routed traffic e.g. service console and virtual machine traffic.  Use E and F for FT and Vmotion traffic that only exists with stacked Virtual Connect switches.  Please see the diagram below showing the configuration for the bottom chassis.  After the networking is set up, assignment for both the Fabric A side and B side needs to be completed.  When finished  you can assign the profile to a blade.  Keep in mind you can also copy profiles to speed up assignment….

This is an overview of a Flex-10 (active/active) stacked HP C7000′s for Vsphere 4.  There will be several posts following suit outlining the setup configuration….

Utilizing HP Virtual Connect and Flex-10 Networking enabled a very clean and efficient enviroment from the aspect of utilizing the two built in network loms both 10gb and providing 10GB uplinks to a core router instead of a cabling mess.

Many resources are available online to get the design that you want although in this example were veering from “the docs” to provide a separate non uplinked Vmotion and Fault Tolerance network for added security.  Since these networks can’t leave the chassis they are more secure.  From a VMware prospective we end up with 4 switches 1 standard vs for the service console allocated at 1gb with 4 uplink (1 to each flex-10 switch), 1 dVS for fault tolerance allocated with 1gb with 4 uplinks (1 to each flex-10 switch),1 dVS for vmotion allocated with 2gb with 4 uplinks (1 to each flex-10 switch), and 6gb to a dVS for VMware networks with multiple Vlans separated out for production and non-production networks.

Core Router Configuration:

Since port channel is Cisco exclusive LACP must be used the upstream configuration is fairly straight forward so we have the following scenario three networks 3 Vlans needed 1 for production VMs (Vlan 95), 1 for development VMs (Vlan 94), and 1 for management network (Vlan96 used for service console) again we won’t be creating uplinks from the Flex-10 switches for fault tolerance or VMotion so no configuration is needed.

Here is the config:

interface TenGigabitEthernet6/1
description HP Flex-10 NIC (left) Port 2
switchport
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 94,95,96
switchport mode trunk
channel-protocol lacp
channel-group 1 mode active
spanning-tree portfast edge

switchport
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 94.95.96
switchport mode trunk
spanning-tree portfast edge