NOT WORKING

Juniper EX4400 – Virtual Chassis not working

October 16, 2023 by Michael McNamara

We made the jump from the EX4300 to the EX4400 this year and while things have been good, we’ve seen a number of bugs and issues with the early software releases.

If you run into issues with Virtual Chassis, my first suggestion is to check the software release.

By default, the QSFP28 ports on the back of the Juniper EX4400 should be setup as “Virtual Chassis” ports for stacking. You can issue the following command to change the configuration if needed;

request virtual-chassis mode network-port disable reboot

The issue I found is that ~ 70% of the time a Juniper EX4400 would fail to see the Virtual Chassis ports (and fail to “stack” properly) if it was running 21.2R3.8 software – the software release Juniper was shipping on switches sold in early 2023. An upgrade to 21.4R3-S3.4 or even the current recommendation of 22.2R3-S2.8 immediately resolves the issue.

I’ve also observed a number of odd PoE/interface issues impacting Juniper MIST Access Points, Kronos clocks along with other assorted PoE devices, such that they receive power but are unable to establish a LINK on the port with either 1Gbps or 2.5Gbps.

I’m currently running 21.4R3-S3.4 in production but we’re seeing a lot of intermittent BFD timeouts which we suspect is a software issue. We’re currently testing 22.2R3-S2.8 in a number of locations.

Cheers

APC UPS NMC stops responding via HTTPS

February 20, 2022 by Michael McNamara

Who doesn’t love a good mystery, I’m no exception. A few weeks back we had an interesting issue pop-up. It was midnight on a Sunday night and PagerDuty started firing off an alert that a UPS in one of our distribution centers had just stopped responding via HTTPS. The UPS was still online responding to both ICMP and SNMP traffic, so the alert was acknowledged and the alarm was paused until it could be reviewed the next day.

The UPS itself was fairly new having been installed just under a year ago. It was an Schneider Electric/APC 3000RT Smart-UPS with an AP9631 network management card. Interestingly enough we just had an issue with a brand new APC 8000SRT Smart-UPS with an integrated AP9537SUM network management card that had essentially started doing the same exact thing a few days earlier. Only that installation was only a few days old when it stopped working. Again ICMP and SNMP worked fine… as did HTTP (if you enabled it).

What was that all about?

After a few hours of troubleshooting and digging I discovered that the self-signed SSL certificate installed on the NMC had expired. Any attempt to connect to the NMC via HTTPS after that point would result in the socket getting immediately closed upon connecting by the NMC. Removing the self-signed SSL certificate and rebooting the NMC caused the self-signed SSL certificate to be regenerated and the problem was resolved. You can remove the SSL certificate by enabling the HTTP server via either SSH or TELNET (will depend on the age of your card as to which one is enabled by default), login in via HTTP go to Configuration -> Network -> Web -> SSL Certificate and select Remove and Apply. You just need to reboot the NMC and you should be able to connect via HTTPS.

1 Year?

The self-signed SSL certificate is only good for one year, after which you’ll need to regenerate it again. The latest version of the firmware/software (NMC2 – v7.0.4) from APC sets the expiration date for all self-signed SSL certificates out to 2035 – not sure if the web browsers will start to complain about that.

Ripple20?

If you haven’t already patched your APC network management cards it might be a good time to take care of that task as well. We had to patch all of our APC and Eaton network management cards that are used throughout our network.

Cheers!

AOL (Verizon) breaks Microsoft Outlook

January 30, 2022 by Michael McNamara

What is going on with AOL and Microsoft Outlook?

I’m a Verizon FiOS customer and was migrated to AOL back in 2017. Within the past 30 days I’ve heard and seen a number of issues with people connecting to their AOL inbox from traditional email clients such as Microsoft Outlook, Thunderbird or even the native email clients on iPhone and Android.

The loving wife had this same issue and I wrongly assumed end user error. You would think I’ve learned by now to not jump to conclusions. It seems she’s not the only person with issues as there are numerous posts on numerous message boards all within the past 30 days with dozens if not hundreds of people reporting the same issue.

The general consensus is that:

Verizon/AOL accounts require an AOL “App Password” to be used as the password for the account configured in Outlook or in any email client (iPhone, Android, Thunderbird, Outlook, etc)

What’s more interesting is that AOL apparently is not blasting out this new feature to all users at the same time because my Microsoft Outlook 365 client continues to work fine while my wife and many others are having to generate an “app password” to get their email flowing again. Some of the posts suggest that if you’ve activated “2-step verification” on your AOL account that you’ll need to generate and use an “app password” to access your email from a legacy email client.

I did find the following article from AOL:
https://help.aol.com/articles/allow-apps-that-use-less-secure-sign-in

The article linked above suggests that AOL is actively blocking clients that it believes are less than secure. Is that because the client is passing the username/password in the clear (unencrypted) in a legacy POP3 connection and not using IMAPS or POP3S?

If your traditional email client stops working it might be more than just a password issue. You might want to try either upgrading your email client or setting up an AOL app password and see if that resolves your issue.

Sign in and go to the AOL Account security page. You can do this by signing on to AOL from a computer.
Click Generate app password or Manage app passwords.
Select your app from the drop down menu and click Generate.
Follow the instructions below the password.Be sure to enter the password into your app without any spaces.Click Done.
Use this app password and your email address to sign in to your email app.