I just recently completed an upgrade of our Cisco Nexus 1000V from 4.0(4)SV1(3) to 4.2(1)SV1(4). Prior to proceeding with the Cisco Nexus 1000V upgrade we had to first upgrade vCenter to 4.1, Update Manager to 4.1 Update 1 and lastly the VMware ESX hosts themselves to 4.1. With all that complete we set out to upgrade the Cisco Nexus 1000V but quickly ran into a few problems.

Pre-Upgrade Script

The Pre-Upgrade Script, a TCL script which checks your Cisco Nexus 1000V configuration for any potential issues, immediately flag our Port Channel configurations in Check 2.

###############################################################################
## COMPANY NAME: Cisco Systems Inc                                           ##
## Copyright © 2011 Cisco Systems, Inc. All rights Reserved.                 ##
##                                                                           ##
## SCRIPT NAME: pre-upgrade-check-4.2(1)SV1(4).tcl                           ##
## VERSION: 1.0                                                              ##
## DESCRIPTION: This script is applicable to all releases prior to           ##
##              4.2(1)SV1(4).                                                ##
##                                                                           ##

...
...
...

=========
 CHECK 2:
=========
Checking for Interface override configurations on Port-channnel Interface ...
###############################################################################
##                           FAIL                                            ##
##                                                                           ##
## List of Port-Channel Interface(s) with interface level configuration(s)   ##

1: port-channel1 has below overrides.
mtu 9000

2: port-channel2 has below overrides.
mtu 9000

3: port-channel3 has below overrides.
mtu 9000

4: port-channel4 has below overrides.
mtu 9000

...
...
...

While originally testing the Cisco Nexus 1000V prior to going into production some 463 days earlier we had played around with enabling Jumbo Frame support within the VM environment. We had set the MTU on the individual port-channel configurations to 9000. Now the pre-upgrade script was telling us that we needed to clean this up and remove any specific configuration from the port-channel and instead apply it to the port-profile configuration. I added the system mtu 9000 command to the port-profile but got a few surprises when I tried to remove the MTU command. The first surprise when I issued the “inter po1, no mtu 9000″ was loosing connectivity to the VM guests on that host. I had to manually restart the VEM on the ESX host with the “vem restart” command from an CLI prompt. The second surprise was that “mtu 9000″ in the configuration had been replaced by “mtu 1500″. That wasn’t going to work so I had to reach out to Cisco TAC who immediately recognized the issue and provided a workaround;

On the ESX host stop the VEM

vem stop

Then on the Cisco Nexus 1000V delete the port-channels and associated VEM (I’ll use the first server as an example assuming there are two VSMs installed)

config
no inter po1
no inter po2
no vem 3

On the ESX host start the VEM

vem start

And sure enough it worked just as promised… recreating the port-channels without the MTU commands. Obviously we had to put each ESX host into maintenance mode before we could just stop and start the VEM.

With that taken care of we started upgrading the VEMs using Update Manager. Unfortunately Update Manager only made it through about 6 ESX hosts before it got stuck. We had to manually install the update VIB on the remaining 12 ESX hosts ourselves. We utilized FileZilla to copy the VIB up to each server and the utilized PuTTY to SSH into each server and manually update the VEM;

[root@esx-srv1-dc1-pa ~]# esxupdate -b ./cross_cisco-vem-v130-4.2.1.1.4.0.0-2.0.1.vib update
Unpacking cross_cisco-vem-v130-esx_4.2.1.1.. ############################################################# [100%]

Installing cisco-vem-v130-esx                ############################################################# [100%]

Running [rpm -e cisco-vem-v120-esx]...
ok.
Running [/usr/sbin/vmkmod-install.sh]...
ok.

With all the ESX hosts upgrade we had to physically reboot the vCenter server to get the currently running VUM task to fail so we could complete the upgrade from the Cisco Nexus 1000V.

Next we launched the upgrade application and before long we had the standby VSM upgraded to 4.2(1)SV1(4). Here’s where we ran into another small scare. After the upgrade of the standby VSM the upgraded VEMs are supposed to re-register to the newly upgraded VSM. We waited about 5 minutes an none of the VEMs had discnonected from the primary VSM running 4.0(4)SV1(3) to the standby VSM that was now running 4.2(1)SV1(4). It was only approximately 15-20 minutes later (while searching Google for some hint) that the VEMs just up and started to connect to the newly upgraded standby VSM.

Cheers!