Field Notice: FN – 72150 – Nexus 9000/3000 Will Fail With SSD Read-Only Filesystem – Power Cycle Required – BIOS/Firmware Upgrade Recommended

The issue was further compounded by some sloppy management, with several switches having unsaved configurations or having crashed and rebooted with unsaved configurations and ultimately inconsistent VPC states. In the short term I ended up deploying the SSD firmware update to all the impacted Cisco Nexus 9000 series switches in my network. I’ll look at performing the recommended software upgrades early next year.

You can setup notifications on the Cisco website to help keep you informed of field notices, software releases and security bulletins.

Anyone else run into this problem?

Cheers!

On Wednesday morning I had two Cisco 5508 Wireless LAN Controllers both crash with a “Bonjour_Process_Task” taking too much cpu: 100% error message. It turns out that this is a known issue (CSCux78464 WLC crashes in Process Bonjour_Process_Task) that is resolved in 8.0.135.1, an engineering release which you need to contact Cisco TAC to obtain. If that wasn’t enough excitement for the morning I quickly noticed that of 120 APs that we usually have connected to the WLC we only had about 70 APs connected.  A quick examination of the debug logs (debug capwap errors enable) showed that multiple APs were failed to join the controller with messages like “Discarding non-ClientHello Handshake or DTLS encrypted packet” and “DTLS session is not established”. A quick call to Cisco TAC revealed that there are built-in certificates into the APs that can expire over time and that’s what had essentially happened. The certificates had expired since the APs had last joined the WLC and now that the certificates were expired they were not able to join the controller. Thankfully there’s a command in the CLI to ignore the certificate expiration;

config ap cert-expiry-ignore mic enable

With that command configured on the WLC the APs starting joining the controller and all was well again.

The field notice from Cisco providing all the details can be found here.

Cheers!

In summary if the switch will continually reboot with a data exception error after the upgrade to 6.1.6 if it was originally configured running 4.x or 5.x software. You can recover from the problem by performing a factory reset of the switch and re-configure it from factory defaults. If you have the ASCII backup I believe you can upload that configuration after factory resetting the switch as opposed to manually re-keying the entire configuration by hand.

If your switch was originally configured running 6.x software then you apparently won’t have any issue upgrading to 6.1.6 software.

I’m not sure how this missed the QA folks… but I’m sure it’s probably hitting the fan right now back at Avaya.

Thanks to telecom116 for bringing the original issue to our attention.

Cheers!

CPU instability issues with Maintenance Release (MR) 4.1.6.0 have been discovered during initial deployment. CPU crash dump events leading to system instability have been reported after the operational SW was upgraded to 4.1.6.0 at different customer site.

There was also an additional note concerning an interaction between Microsoft’s Network Load Balancing (NLB) when running in multicast mode on 4.1.6.0 software code that might result in a CPU crash on the ERS 8600.

Nortel is advising that 4.1.6.0 software has been pulled from their website and replaced with 4.1.6.3 software. They are also advising that 4.1.7.0 software will be available in August 2008.

While I haven’t personally seen this problem I only have 1 switch running 4.1.6.0 software out of the 37 ERS 8600 switches we have in production.

Cheers!