I recently stumbled into yet another interesting issue that turned out to be a bug in the SSD firmware of some Cisco Nexus 9000 Series switches. We had performed an upgrade in two of our Data Centers just over 3 years ago using the Cisco Nexus 9000 Series product line providing a 10/40Gbps network. Within the past week we had several of those switches crash and reboot themselves. Upon further investigation I found some switches that didn’t crash or reboot themselves were running with a read-only file system. It turned out that this was a known bug that had been identified by Cisco earlier this year.
Field Notice: FN – 72150 – Nexus 9000/3000 Will Fail With SSD Read-Only Filesystem – Power Cycle Required – BIOS/Firmware Upgrade Recommended
The issue was further compounded by some sloppy management, with several switches having unsaved configurations or having crashed and rebooted with unsaved configurations and ultimately inconsistent VPC states. In the short term I ended up deploying the SSD firmware update to all the impacted Cisco Nexus 9000 series switches in my network. I’ll look at performing the recommended software upgrades early next year.
You can setup notifications on the Cisco website to help keep you informed of field notices, software releases and security bulletins.
Anyone else run into this problem?
Cheers!