vSphere 6 Upgrade Fails – “The Upgrade contains the following set of conflicting VIBs:”

I was upgrading an ESXi 5.5 host for a client and ran into some “Incompatibility” errors. They had a mix of Dell server hosts, but three of them were Dell R715’s and all three were getting upgrade errors. I first tried the update using VMware Update Manager (VUM), and made sure I was using the Dell Customized ISO, which includes Dell specific drivers. (You can download the ISO here: http://goo.gl/3UOeNV ).

After adding the ISO to a new upgrade baseline and scanning the host for updates, I was the following errors:
Compliance State: Incompatible
The upgrade contains the following set of conflicting VIBs:
Mellanox_bootbank_net-mlx4-en_1.9.9.0-1OEM.550.0.0.1331820
Remove the conflicting VIBs or use Image Builder to create a custom upgrade ISO image that contains the newer versions of the conflicting VIBs, and try to upgrade again.

Attempt to continue the upgrade and dismiss the errors, resulted in upgrade failure. Any attempt to upgrade via the CLI also failed.

So what is the problem and how do you fix it?

The problem is with incompatible drivers that are currently on the host. Drivers that aren’t supported by ESXi 6, and drivers that aren’t included in either VMware’s or Dell’s ISO.
This particular VIB is a Mellanox Infiniband HBA, which probably most of us seeing this error do not use.

To remedy this issue and proceed, we need to remove those drivers from the host.

First, enable SSH on the host that has the issue
Next, SSH into the host and run the following commands:

~ # esxcli software vib list | grep Mel
~ # esxcli software vib remove -n net-mlx4-en
~ # esxcli software vib remove -n net-mlx4-core
~ # reboot

It may take a min or so after running commands two and three, but it should complete successfully. After rebooting the host, proceed to upgrade via Update Manager or CLI.

After I completed the above instructions and scanned my host again with Update Manager, it found one more incompatible VIB that I had to remove on all three servers.

Compliance State: Incompatible
The upgrade contains the following set of conflicting VIBs:
VMware_bootbank_xhci-xhci_1.0-3vmw.550.3.78.3248547
Remove the conflicting VIBs or use Image Builder to create a custom upgrade ISO image that contains the newer versions of the conflicting VIBs, and try to upgrade again.

I was able to fix this in the same manner I did the previous VIB:

~ # esxcli software vib list | grep Mel
~ # esxcli software vib remove -n xhci-xhci
~ # reboot

Upgrade VMTools on Linux Appliances and OS

Upgrading VMTools on a Linux OS isn’t as convenient as the Auto-Upgrade for Windows. Each flavor of Linux has their own CLI commands for upgrading VMTools. I have listed a few Distros along with their corresponding CLI Commands.
One note to remember- Most Appliance based VMs have VMTools baked in, and are usually out of date. Upgrading VMTools (or Virtual Hardware for that matter) on these appliances may “break” the VM. I recommend checking with your appliance vendor to see if they have any updates for the appliance, or to verify that VMTools can infact be upgraded.

SUSE LINUX 11

  1. Mount VMTools installer on VM (Right Click VM and choose Guest – Install VMware Tools)
  2. Launch YaST | Software | Software Management
  3. Change Filter to Patterns, and make sure c++ compiler is installed.
  4. Right-click the VMWare guest in the VMWare client, and click Install VMWare tools
    • This may also be under a Guest tab after right-clicking on the guest VMWare
  5. Close YaST
  6. Type mount /dev/cdrom /media
  7. Type cp /media/*.tar.gz /tmp
  8. Type cd /tmp
  9. Type tar -zxvf VM*.tar.gz
  10. Type /tmp/vmware-tools-distrib/vmware-install.pl –default
  11. Type init 6 to restart the server
  12. Type /etc/init.d/vmware-tools status to make sure it is running
    1. Right Click VM – Guest – End VMware Tools Install

 

REDHAT/CentOS

  1. Mount VMTools installer on VM (Right Click VM and choose Guest – Install VMware Tools)
  2. $ yum -y install kernel-devel gcc dracut make perl eject
  3. $ mount /dev/cdrom /media
  4. $ tar -zxf /media/VMwareTools-*.tar.gz -C /tmp
  5. $ /tmp/vmware-tools-distrib/vmware-install.pl –default
  6. $ rm -rf /tmp/vmware-tools-distrib
  7. Right Click VM – Guest – End VMware Tools Install

 

Ubuntu

  1. Mount VMTools installer on VM (Right Click VM and choose Guest – Install VMware Tools)
  2. sudo mkdir /mnt/cdrom
  3. sudo mount /dev/cdrom /mnt/cdrom or sudo mount /dev/sr0 /mnt/cdrom
  4. ls /mnt/cdrom
  5. tar xzvf /mnt/cdrom/VMwareTools-x.x.x-xxxx.tar.gz -C /tmp/
  6. cd /tmp/vmware-tools-distrib/
  7. sudo ./vmware-install.pl -d
  8. sudo reboot
  9. Right Click VM – Guest – End VMware Tools Install

Dell DPACK 2.0

For those not familiar with the “Dell Performance Analysis Collection Kit” (DPACK), it is a pretty incredible tool that allows you to visualize the current storage and server workloads from the perspective of the host. It is a great tool for planning Capacity and I/O requirements, and can even be used to troubleshoot problems with your storage and storage network. DPACK measures the following:
– Disk I/0
– Throughput (Which is more important than I/O when we’re talking FLASH)
– Capacity
– Memory Consumption on your Servers
– CPU Utilization on your Servers
– Network Traffic
-Queue Depths

Version 2.0 allows real-time analysis statistics that you can view, instead of having to wait the 24 hrs you were required to wait and let it run previously. Version 2.0 gives you better views into your data for additional insight, and presents this data in a form that Executives can appreciate when you go to them with a PO Request.

Some other things to know about DPACK 2.0 are:
– Uses HTML 5 for viewing real-time data in a browser
– Generate PDFs of data collected to present to management
– DPACK compresses analyzed data and transmits to server every 5 mins
– Uses secure SSL on port 443
– DPACK will continue to run and collect data in the event it loses ability to upload
– DPACK isn’t performance impacting and can be (and should be) run during business hrs

So how do you get started with DPACK 2.0? Its best to call up your local Dell Storage Team and discuss DPACK. Netwize, is a great resource for any Dell Data Center needs, helps customers run DPACK all the time and are a great resource for you to reach out to. I work for Netwize so please feel free to reach out to us and we will get you set up.

VMware Update Manager (VUM) Repositories – HP, Dell, IBM

VMware’s Update Manager is a pretty convenient tool that allows you to patch your ESXi hosts with the latest security, feature, drivers and bug patches. By default, VMware Update Manager downloads these patches from its own repository, but sometimes (usually) doesn’t have the latest drivers and patches for host servers from their actual manufactures (Dell, HP, Cisco, IBM). Most these vendors offer a repository that VUM can use to download drivers from those manufactures. Here is how you do it.

Assuming you have Update Manager installed, in the Windows vSphere Client (Web Client can’t use VUM yet)
1. Go to the “Home” screen.
2. Toward the bottom of the screen, click “Update Manager

3. Then click on “Configuration” on the Top Tab
4. “Download Settings” on the left side,
5. And finally, click “Add Download Source” above the repository window.

Here you can add URL’s that the manufactures have provided to download the latest drivers and patches. VMware Update Manager will periodically check for new updates from these URLs, and download them automatically. Not every manufacture provides this feature, but here are the ones I could find.

HP Drivers
HP Management Tools
Dell
Brocade 1
Brocade 2

Change VMware vCenter Appliance’s Default Password, or Else…

For those running VMware’s vCenter appliance (if not, it should be something to consider in vSphere 6), you may run into an issue where you cannot authenticate using the default appliance credentials.
Username: root
Password: vmware

You have 90 days to change this password or else you will see this error when trying to login to the appliance:

As a note: Once you have changed the default password, you can choose to have your new password NOT expire. This is done in the appliance web portal- https://applianceIP:5480

For those that are facing this issue, there is a solution! (Besides blowing the VM away and starting from scratch). This can be done using some commands, but needs to be done from a “LIVE CD”. A Live CD is a bootable ISO (or actual CD) you can download and boot into. This allows you to run a temporary environment that has access to the actual installed OS.  Any Linux Live CD will do, but I prefer Kali Linux or Ubuntu. You will need “console” access to your vCenter appliance, which you can access from the vSphere Web Client. Just connect to the actual host where your Appliance VM resides.

– Mount the Live CD to the VCenter Appliance VM and boot into the ISO/CD.

– Next, bring up a Terminal or Shell and enable root user:

su -

– Then we need to mount the vCenter Virtual Appliance’s root parition

mount /dev/sda3 /mnt

– We need to /mnt/etc/shadow file to disable the account lock. Use the Vi editor to make the edit.

vi /mnt/etc/shadow

*When the root password is expired there should be an x in front of the password string.

– Delete the “x” character in front of the password string. (The password string is the text after “root:“. Then save the changes in Vi editor (ZZ is the Vi Editor SAVE command, then quit with :q command).

– Unmount the LiveCD and reboot the vCenter Appliance VM. Now you should be able to login with the default/last root password you had set.

– If you don’t want the password to expire, you can login to the appliance (now that the password has been enabled), click on the “Admin” tab, and check the box to NOT expire the password

Patch and Update VMware ESXi via SSH CLI

Most clients using VMware patch and update their vSphere Hosts using VMware Update Manager or VUM. This provides an easy GUI to find needed updates and apply them to your ESXi Hosts. Once in awhile, VUM has issues patching hosts. And to use VUM, you need a valid licensed vCenter. So those of you using the Free ESXi or need to patch hosts not registered to your vCenter, the easiest way to patch your hosts is through the SSH Command Line (CLI). I will run this process down, step by step.

– Login to your free MyVmware.com account and download the latest patches
http://www.vmware.com/patchmgr/download.portal

– In the portal, select the ESXi (embedded and installable), and select your version of ESXi you need to patch. Then download the latest update. The patches are cumulative, so the latest update will include all previous patches as well.

– Shutdown your VMs using the Windows vSphere Client and put the host into maintenance mode.

– Enable SSH on the host by selecting the host in the vSphere Client, clicking on Configuration, Security Profile, and click Properties in the top right hand corner. Open SSH and click the Start button.

– Using the vSphere Client, browse to a local Datastore and upload the downloaded patch to the root of the Datastore, (or a folder of your choosing).

– Using Putty or some other SSH tool, type in the IP address of your host, and login with the “root” user.

– Now on to patching the host. Type the following command, replacing this example path with the patch of where you uploaded the patch (in zip format).

esxcli software vib update -d /vmfs/volumes/<your_datastore>/<name_of_the_patch_you_uploaded.zip

– Wait a few minutes, after which you should see a bunch of text showing the status of the update.

– Finally, type “reboot” and you should be all updated after the reboot!

 

Running Dell DPACK longer than 24hrs

If you have ever used the DELL DPACK utility to analyze your storage, you’ll find that the application only gives you the ability to scan for 24hrs. Dell does this because they claim “statistically” the DPACK results don’t vary much from any particular day to day. Having run hundreds and hundreds of DPACK scans for many customers, I find that more often than not, the results from various days of running are different enough to warrant longer monitoring times. My advice is to run your DPACK for 3-4 days. And here is how you do it

First download the DPACK tool from http://dell.com/dpack
Extract the Software
Open a Command Prompt and change directory to the extracted DPACK folder
Run the following command: dellpack.exe /extended
CMD_Prompt__BLUR2_.png

Now you should be able to change the monitoring duration:
Time_Duration.png

VMware vSphere ISCSI Port Bindings – You’re Probably Doing it Wrong

As a Consultant, I have the pleasure of seeing a lot of different operating environments from a variety of customers. At a high-level, most customers have the same data center infrastructure (Servers, Storage, Running Virtualization, etc). Although the configurations of these environments vary, I see one configuration mistake made by many of these customers – “ISCSI Port Binding”.

For those unfamiliar with ISCSI Port Binding, Port Binding binds/glues the ISCSI Initiator interface on the ESXi Host to a vmknic to allow for ISCSI multipathing. Binding itself techincally  doesn’t  “allow multipathing”, just having multiple adapters can do that. But if you have Multiple Adapers/VMkernel ports for ISCSI used in the SAME subnet/broadcast domain, it will allow multiple paths to the ISCSI array that broadcasts one single IP Address.

Why do I need to bind my Initiator to a VMkernel anyway?
When you have multiple ISCSI Adaptors on the same subnet, there is really no control on where data flows or how to control data broadcasts of the adapters. You literally flood that network with rouge packets.
* Note: I am trying to make this easy to understand for those that don’t have a deep technical experience on this subject. And in doing so, I am only telling half-truths here to keep things simple. Don’t call me out on this :)

When should you enable ISCSI Port Binding?

ISCSI Port Binding is ONLY used when you have multiple VMKernels on the SAME subnet.

Pictured above, you can see there are multiple VMkernel ports on the same subnet and broadcast domain. You MUST use port binding! If you do not, you may experience the following:
– Unable to see ISCSI Storage on ESXi
– Paths to storage are reported as Dead
– Loss of Path Redundancy Errors
ISCSI Port Binding bypasses some vSwitch Functionality. No Data Path, No Acceleration.
Array Target ports must reside in the same Broadcast Domain & Subnet as the VMkerel port
All VMkernel ports used for ISCSI must reside in the same broadcast domain & subnet
All VMkernel ports used for ISCSI must reside in the same vSwitch

When should you NOT enable ISCSI Port Binding?

Do not enable Port Binding if Array Target ports are in a different broadcast domain & subnet
ISCSI VMkernel ports exist in different broadcast domain, Subnet an/or vSwitch
Routing is required to reach the array
If LACP/Link Aggregation is used on ESXi host uplinks to the pSwitch

In the above example, you should NOT use Port Binding. In doing so, you may experience:
– Rescan times take longer than usual
– Incorrect number of paths per device are seen
– Unable to see any storage from the array

So why do I say you are probably doing it wrong? Most storage arrays use the second example as a best practice for mulitpathing to the array. Most customers follow those best practices and use two VMkernel Ports on different subnets to connect to their arrays. But most people still enable port binding!
If you are guilty of this, you can easily remove the existing Port Bindings. Doing so will cause a temporary loss to your storage, so make sure all VMs are shutdown, and you have a maintenance window.

Now you know!

 

 

Edit VMware 5.5 Virtual Machines without Web Client

VMware was pretty sneaky about its last Virtual Hardware update (vmx-10) in the vSphere 5.5 release. After updating your Virtual Machines, you could no longer edit the VM’s settings using the vSphere Client.

 

You were forced to use the Web Client to edit settings, but the Web Client isn’t available to standalone hosts. It is only available for environments with a licensed vCenter. (Test/Dev environments were out of luck). I assume after the frustrated customer uprising that took place on web forums, VMware has finally given us a solution. Customers just need to upgrade their vSphere Client to 5.5u2.
https://my.vmware.com/web/vmware/info/slug/datacenter_cloud_infrastructure/vmware_vsphere/5_5

One thing to mention though, they still have stripped customers of taking full advantage of editing their VM Settings. You are only able to edit hardware version 9 and 10 VMs, but only on the feature level of hardware version 8. (For real?)

Here are the list of properties that are editable

      • Increase RAM
    • Decrease RAM
    • Change network port group
    • Remove devices
    • Increase vCPU
    • Decrease vCPU
    • Mount ISO
    • Increase disk space
    • Set a limit
    • Remove a limit
    • Set a reservation
    • Remove a reservation
    • Edit advanced settings