I was recently asked how difficult it was to stack a pair of Avaya Ethernet Routing Switch 4500 Series switches. I responded that it wasn’t hard at all and so began my downfall. While it isn’t very hard (honestly it isn’t) there are a few gotchas that you need to look out for and I ran into everyone of them as minutes turned into hours and for whatever reason I just couldn’t get my mojo going.
In the hopes of avoiding the same pitfalls again in the future I’m going to document the gotchas so that I might remember to read this post and plan accordingly in the future. I’m always happy to share so I hope this can help others out there so that their 30 minute change doesn’t end up taking all night long.
This first tip will apply regardless of the number of switches you already have running, be it 1 or 7. The remaining items will only really have a bearing if you are making a standalone switch a stack for the first time.
- The software and firmware code must match before you do anything. You should either upgrade or downgrade the switch(s) such that the software (agent) and firmware (boot or diagnostics) code is identical on all the switches you are going to stack together. Performing this task prior to stacking the switches will save you a lot of time and headaches. You can temporarily connect the new switch to your network with a temporary IP address and either upgrade or downgrade it as necessary. When you are finished make sure to factory reset the switch before trying to connect it to an existing switch or stack.
- Make sure that only 1 switch has the base selector set near the cascade/stack ports. The top switch in the physical stack will traditionally be set to base but you should verify that the selector below the cascade/stack ports is set to base on what I’ll call the “master” switch.
- Performing the actual stacking is pretty straight forward. If there is a stack already in place just break the stack (usually at the bottom) and connect the new switch to the last switch in the stack and to the top most switch in the stack. You need to be careful of the stack cable lengths, the default provided cable is 1.5m but you can order cables up to 5m in length if necessary. If this is a new stack you’ll need to retrieve the stacking cable that came with the original switch (I’m sure you still have it somewhere right?) and you’ll need to use both cables to complete the stacking ring. You can find some additional information and notes in this post regarding the physical changes.
- Stack IP Address / Switch IP Address. If you are making a standalone switch a stack for the first time you’ll notice that the switch will stop pinging once it successfully stacks. You’ll need to use the serial console interface to change the switch IP address and stack IP address. You need to essentially move the switch IP address to the stack IP address field. You can blank out (0.0.0.0) the switch IP address. If the switch had the IP address of 1.1.1.1 here are the commands I would use to move that IP address to the stack interface;
4548GT-PWR(config)#switch ip address 0.0.0.0 4548GT-PWR(config)#stack ip address 1.1.1.1
- Username / Password. When you stack a pair of switches for the first time you’ll notice that the username and password reverts to the factory default settings. If you haven’t changed the stack username it defaults to “RW” with no password. If you having configured a special username or password you need to reconfigure that once the switches are stacked.
4548GT-PWR(config)#cli password ? read-only Modify read-only password read-write Modify read-write password serial Enable/disable serial port password. stack Modify stack passwords. switch Modify switch (stand-alone) passwords. telnet Enable/disable telnet and web password.
Those are a few of the gotchas that can turn a quick and relatively easy change into a fiasco.
Cheers!
Changing the base switch “usually” requires a reboot to take effect. (YMMV; I’ve always rebooted the base changed switch just to make sure).
I would agree with you, although I haven’t tested it in a long time now.
Thanks for the comment!
The “blink-leds” CLI command can be useful if your units number doesn’t match physically and logically.
I have yet to experience a defective unit replacement procedure with the 45xx series, but it was a little bit tricky few years ago (470T). Not sure if the stack detects the replaced unit and assigns the correct unit number or not (wasn’t the case before, you had to delete the defective unit and than renumber the replacement unit). And I had to reconfigured it almost every time. The new AUR feature is still untested for me :)
Make sure to check the RELEASE NOTES before upgrading, if it’s been a while since you upgraded, you might not be able to update to the lastest release right away. Been caught once or twice and it’s no fun when a stack can’t boot and you have to save it using a laptop connected to the stack with a tftp server on it.
I would agree regarding AUR (Automatic Unit Replacement)… my experience has been hit or miss so much so that I usually avoid it entirely.
However, to be fair I haven’t tested it in a while now and it might have improved.
Thanks for the comment!
I put the base switch to BASE when I only have 1 unit, that way if I need to add a second unit later, no need to reboot.
That’s a great tip!
The AUR works great, but never for the Base-switch, as all others get their config from the base. It happened to me once with a failing base stack. Putting back the txt-version of the config doesn’t work for a single switch. If you try to put back the txt-config on a stack, the whole stack reboots (And that’s not fun on an iSCSI-blade-rack, as it cannot do without connectivity for more than a minute).
The only way is using the binary-config-file, if you have one. But I’ve had more problems than i’d like to remember with binaries, and i’ve never tested this in a live environment.
I had too many problem with binary-config-file too. Just dropped the thing long ago and mostly use text-config-file, but you still can run into problem and end up doing half the work manually.
Hi Danny,
It’s been my experience that the ASCII configuration file will work as long as you factory reset the switch before trying to upload (or cut-n-paste) the ASCII configuration file. When I run backups these days I grab a copy of the BINARY and ASCII configurations just to avoid any potential issues if the BINARY configuration doesn’t work.
Thanks for the feedback regarding AUR, I might head to the lab to check it out.
Cheers!
If you want to stack some switches which having set different Passwords, the Switches will not join the Stack.
I ran into that security feature recently…
Hi Maik,
That’s why you should factory reset the switch you are going to add to the stack, this way you know there’s no configuration on it before you stack it with your existing (production) switch or stack.
Thanks for the comment!
And why stack switches again?
Admittedly, my environment doesn’t have very large wiring closets – the largest is maybe 150 ports – but I still prefer straight uplink ports between closet switches.
Closet switches typically have next to zero communications between access ports, since everything lives on servers past the uplink ports, so the ridiculously fast backplane via stacking cables isn’t an advantage.
Switches only reboot due to power failure (in which case PCs are already down) or a scheduled service outage.
Switch configs for a 24 or 48 port switch are small and manageable, and can be automatically backed up using some kind of third party script or application.
I imagine that anyone with a network large enough to consider stacking also has a configuration management and network monitoring solution in place. Given the fact that so much can be automated with these tools, what advantages are left for only having one IP address to manage a switch stack?
I guess that I’m not sold on the stacking solution for a wiring closet, and I don’t have the server density to warrant investigation in a computer room or DC.
Hi MisterAG,
I’m a little confused with your comment… in a hub and spoke topology why in the world would I not stack switches? As you mention it’s all about over-subscription and making sure you size the uplinks/downlinks appropriately. If I have 192 ports connecting to thin clients then an over-subscription ratio of 96:1 (192Gbps:2Gbps) is quite acceptable. If I have 10 radiologists performing utilizing a cardiology PACS system then that over-subscription ratio probably isn’t going to cut it since PACS systems can chew through bandwidth at an alarming rate depending on the sizes of the DICOM studies.
Here’s a quick example based on one of my sites…
I have 80 stacks/switches at this location spread across ~ 55 ICRs (Intermediate Communications Rooms). On average every stack has ~ 4 switches which equates to 320 switches all total in that network. I have 2 ERS 8600s in the core each with capacity for ~ 108 fiber downlinks. I have some closets with multiple stacks and some stacks with 4 fiber uplinks (generally each stack has 2 fiber uplinks spread out across multiple switches -DMLT- in the stack for redundancy) depending on the size of the stack and the equipment connected to the stack. I currently have 80 IP addresses to monitor, configure, upgrade, etc. I do it all with homegrown Perl scripts and some help from a HP Open View as an alarm platform to receive SNMP traps and fire off alarms.
If I didn’t have the ability to stack switches… I would have 320 switches to individually monitor, configure, upgrade, etc. I’d need 640 uplinks in my core to accommodate a single uplink from each switch to each of my cores.
In a small network with a few switches I can see your point but in a network of any significant size I couldn’t imagine the effort to manage all those switches.
Thanks for the comment!
I can’t imagine running my network without stacks. For me, it would almost be like going back 15+ years ago when vlan didn’t exist much :)
Having 50+ wiring closets here, a few stacks of 5530s, but mainly 4550/470-48T/2550/4850 stacks . Most stacks are 2-5 units, but got a few stacks of 6 or 7 units. Much easier to manage having only 1 IP for the whole stack. Also, I’d either need to connect a fibre in each unit or cascade them (ugly and not efficient for me) as I can have a stack of 5 units, but only two uplinks spread over 2 of those units. IMHO, stacking gives a much more reliable network as you can use MLT to connect to two separate units and if one of those unit fail, your link is still up.
The whole documenting aspect seems easier with stacks.
I completely agree…
I’m currently replacing equipment in a facility where the former consultants daisy chained switches in the closet and it’s reaffirmed my belief that we’re doing it the right way and have been for years now.
personally, I never liked the term “Base”. Because the word it implies “low”, “foundation”|, “bottom” . And although you can make the base switch any switch in your stack, most people new to nortel/avaya kit instinctively think that (or configure) the base switch as the bottom switch. But as we all know you can make the base switch any switch in the stack depending on the stacking cable cascade and the config.
however, when you load JDM or EDM. it always shows the “base” (switch #1) as the Top switch in a stack. Regardless of the logical layout in your racks.
as a result. To avoid confusion and maintain continuity I always make my stacks with the Base as the Top 1st first switch, then work my way down.
1 (base)
2
3
4
5
6
7
8
I have been burnt too many times when I see a stack configured in the following order.
4
3
2
1 (base)
or even worse.
2
3
4
5
1 (base)
Hi Paul,
That’s a great point… don’t assume they are stacked in the correct order. You should verify the order of the stack.
The command above from the CLI will cause the LEDs on the switch to blink allowing you to confirm the order of the switches in the stack.
Cheers!
The second numbering scheme is the actual method that they were originally designed for. The Baystack scheme was to always stack up just like an elevator. The reality is that natively everyone reads top to bottom so we expect numbers to build down not up.
This is atleast the folklore that I was told by some old Bay guys when I started at Nortel.
Also of note, using AUR nullifies the necessity to match code revisions before stacking. It is however something that was required in the Baystack [and Ethernet Switch] 4xx/3xx series.
The switch to stack settings in terms of passwords is not that it changes the password, there are just different passwords for the switch and stack designation. If you destack the unit you will see that your old passwords are still there.
All of these things in consideration should lead you to understand that there are some peculiarities in a stack of two failure scenario. This is countered by the forced-stack mode which should be utilized to get around this changing from stack back to standalone switch.
Finally, someone noted about the failure of the base unit and replacing it. There are procedures, but the quickest and foolproof method for base unit replacement is to change out the configuration with the unit out of the stack and then add it in, thus creating a new stack with the same configuration. Additionally, make sure that the code revision of the new base unit is the same as the rest of the stack as with AUR, it will think that it needs to adjust the code level of the rest of the stack.
yup, there is similar show command. It will blackout ports 1-12. and only light up ports 1 through 8 to ID the switch number.
when i add a new switch with a different sw to the stack, the switch upgrades automatically , reboots and then makes part of the stack. Is this a bad procedure ?
Hi Dan,
I would recommend that you upgrade and factory reset it prior to adding it to any stack. In theory (and sometimes reality) the auto-upgrade feature works but it’s only available on the later software releases and I’ve seen it choke on a few occasions. Now that I’m thinking of it I don’t believe it will auto-upgrade the FW version only the SW version.
Cheers!
I always upgrade (fw and sw) to the same version of the target stack and factory reset the new unit(s) before adding/replacing any unit in a stack. Check the Release Notes for upgrade path if you don’t want any bad surprises.
It’s probably an old habit I have since the Baystack days and maybe it isn’t necessary nowadays, but really, it only takes what, 5 minutes…. I hate having to take my laptop on the field because I can’t do anything remotely to fix it. 5 minutes to be almost 100% sure it will works once my guys take it out there… totally worth it.
Dear,
i have the 4850GTS-PWR stack switches there,but now 1 switch is not work,i want to replace the switch ERS4550T-pwr,is it work in stack.please let me know.
can we configure stack on 2 different module like ( 3549GTS with 48 ports and 3524GT + PWR with 24 ports ) ?
Hi Francois,
I believe you can stack any 3500 model together.
https://www.avaya.com/en/documents/avaya-ethernet-routing-switch-3500-series-lb7028.pdf
Cheers!
show stack-info will also light the led’s. One thing I wonder about is switch numbers. In the Cisco world the switch remembers it’s number. It can join a new stack and will keep it’s number if that number is not in use. If run standalone it will run as it’s number. Each switch has it’s own copy of the config. If it joins a new stack it will default all it’s interfaces and download the global config from the master.
The other day I stacked a bunch of 4950GTS on a table in prep for putting into a rack. Once in the rack I noticed the switches were not numbered according to the cascade, but were numbered top to bottom with base at the top as follows. 14235. I renumbered them but, this leads me to believe that 4950 keeps it’s switch number as well. Or perhaps they were confused because I did have a stack issue when I powered them up. Due to a mishap the 2nd switch from the top had a bad stack port and so was not connected to the base and there was a bad stack cable between the bottom and the 4th. I replaced the 2nd switch first and then the bad cable. Once done the base lights stopped blinking and the base light came on on the base unit.
I had test practiced stacking with three Avaya 4826GTS switches that were extras. Two were originally stacked and I was adding another.
I read your write-up and the practice run was successful.
However, it was several weeks before I actually tried to add two of the extra switches to an existing stack of two and I screwed the pooch by not changing the selector switch from BASE, and I was adding the two additional switches at the same time.
I don’t know why or what I was thinking at that brain fart moment but I caused a big problem before I could stop it.
I returned the original stack to its former configuration by removing the two switches that I was adding but something went bad or wrong where the two number one ports that connect to an Alcatel fiber switch stopped communicating with the Avaya 4826GTS at the other property.
I had to contact the person that installed them to look at my problem and he was able to get the connection back up. He said that it is a problem with multilink trunk configuration of bonded or lag settings of port(s) and has to get with Cox fiber team to correct the settings that were there before.
I really made a big mistake from what I did compared to my practice run with the extra stacked pair and adding a third switch to the stack.
What do you thing happened when I attached the two additional switches to the existing stack simultaneously AND forgetting to move selector switch from base?
It sounds like the switches in your stack somehow got re-numbered… the “BASE” switch would not cause that. You had two different problems.
Cheers!
Hi, highly useful. What is the difference between switch ip and stack. here we left the switch ip with 0.0.0.0.
Hi Sabarivasan,
You use the “stack” IP address when the switch is in a stack configuration with other switches. You use the “switch” IP address when the switch is used as a standalone switch as a simple Layer2 switch.
Cheers!