I’ve been working with Motorola and Nortel for over the past 9 months troubleshooting an issue that was causing the Nortel wireless phones (2210, 2211, 6120, 6140) to reset while the phone was idle. We eventually traced the problem to a buffer overload issue on the AP300 due to the extreme chattiness of the Spectralink Voice Priority (SVP) and UNIStim protocols and the prolonged power save polling (1.5 seconds) of the Nortel wireless phones. Motorola just released v22.214.171.124 and v126.96.36.199 software for the RFS7000 and WS5100 respectively that resolves this problem by increasing the buffer space on the AP300 allocated per (voice) mobile units. Thanks to Nortel and Motorola for their diligent work in tracking down this “needle in a haystack”.
It was a challenge to understand all the different heartbeats, timeouts and protocols that were in play between the handset and the Nortel 2245 wireless gateway and ultimately the Nortel Succession Signaling Server. With any Nortel IP phone running a UNIStim protocol there is a watchdog timer on the phone that counts down from 200 seconds. The watchdog timer must be reset by a watchdog reset (heartbeat) message that gets sent out from the Nortel Succession Signaling Server. This watchdog reset gets sent every 30 seconds. If a handset, remember now any Nortel IP handset that is running a UNIStim protocol such as the i2002, i2004, 1120e, 1140e, 1150e, 2210, 2211, 6120 and 6140 misses too many of these heartbeats the phone will reset itself usually displaying the message “watchdog timeout” indicating that the watchdog timer has reached zero and the phone is attempting to recover from the problem by resetting itself. With the Nortel 2210, 2211, 6120 and 6140 you also have the SVP heartbeats and timeouts to worry about.
If you have some IP phones that are generating “watchdog timeout” message your probably loosing packets somewhere in your network. With that said I would advise anyone with such a problem to immediately contact their voice reseller and make sure their Succession Call Server and Signaling Server have the latest and greatest DEP (patches) list. Once that’s complete you’ll need to go about the task of isolating the possible locations where you could be dropping packets. If it’s a wired IP phone then the problem is much easier to troubleshoot and isolate. If it’s a wireless phone then you’ll have a few extra steps. You’ll obviously need to make sure that you have QoS (DiffServ) up and working within your environment and you’ll need to make sure that you have SVP support enabled on your wireless infrastructure. SpectraLink (recently acquired by Polycom) actually has a library of documents to help customers configure their wireless infrastructure properly to support the SpectraLink handsets.
Correction: August 19, 2008
The watch dog interval is actually 200 seconds long and not 120 seconds as originally posted.
Update: August 24, 2008
It would seem that this article has generated a lot of interest including several inquiries by Nortel. So I thought I would try to add some additional explanation to help more clearly describe the problems and experiences I’ve had the Nortel 2211 and 2210 wireless handsets. I won’t rewrite the original because I don’t think there is anything wrong with it, other than perhaps missing some attention to the specific details.
The Motorola WS5100 v3.x and RFS7000 v1.1 was technically broken for anyone using the Nortel 2211/2210/6120/6140 wireless handsets. The phones would often reset while idle, because of a buffering issue on the Motorola AP300 access port. These problems have been resolved (as far as my testing indicates) in the Motorola WS5100 v3.2 and RFS7000 v1.2 software release. Through our troubleshooting of this problem we learned a great deal about the Spectralink Voice Priority protocol and the UNIStim protocol. In short the Nortel wireless handsets will go into PSP (Power Save Polling) for approximately 1.5 seconds, during that time the wireless handset turns off it’s radio to help save power and preserve the battery life. The problem occurs while the phone is idle because of the PSP mode, this is why no problems are ever reported while the phone is off-hook and actively being used. While the wireless handset is in PSP mode the wireless network is responsible for buffering any packets that are sent to the handset. The SVP protocol and UNIStim protocol can generate a lot of packets causing the wireless network to discard some packets while the phone is in PSP mode. These discarded packets can, depending entirely on the timing, cause the phone to either reset or the phone to be unregistered from the Succession Signaling server.
I’ve been asked by quite a few people what can be done to help alleviate any potential issues?
- The wireless infrastructure should be configured to support the SVP protocol
- QoS (DiffServ) should be set to “Trusted” on every Ethernet switch port that will be used to connect the different equipment (Succession Signaling Server, Succession Voice Gateway Media Card, 2245, wireless infrastructure)
- Design the wireless infrastructure so there is at least -60 dB of signal available and no more than 7 wireless handsets connected to a single access point/access port.
With all that said Nortel has literally just released v97.072 software for the Nortel 2211/2210 wireless handsets. While the release notes don’t seem to indicate any changes that are specific to “watchdog” issues it might be worth giving it a shot.
Update: Friday September 12, 2008
I’ve placed a copy of the Nortel document WLAN IP Telephony Installation and Commissioning (v3.3) on my website. This document should be a great help to many folks that are having issues with Nortel 22×0 and 61×0 wireless handsets.
Wim Coenen says
The original author has requested that this comment be deleted.
Michael McNamara says
I’ve removed your phone and email address from your comment, don’t want to contribute any more SPAM to the world. I don’t have any specific Nortel contacts for my specific problem. I opened a trouble ticket with Nortel and Motorola and worked through the process of getting the ticket escalated appropriately with both companies. The problem with Motorola was specific to changes they made in the AP code between their 2.x and 3.x software release so I’m not sure the same applies to your problem.
Have you configured the Trapez wireless switch equipment to support SVP as per the configuration recommendations put forth by Spectralink (now Polycom)?
Watchdog timeouts are a sure sign of loosing packets between the Nortel Succession Call Server and the wireless handset. The problem is locating where those packets are being lost, wired network or wireless network.
Thanks for the comment!
Wim Coenen says
Yes we’ve setup the Trapeze equipment for SVP support and we are using the 2245’s in the new clustermode.
Is it possible to pass me (on my email adress) the name(s) of the Nortel technician(s) who handled your case. This might help us to speed up things. Also Nortel Europe contacted me with the question to remove our conversation from your blog. That’s OK with me, you helped me anyway. If you want I can rewrite my first reaction without upsetting anyone.
Ray Jenkins says
Our facility also experienced issues with watchdog timeouts using the 6k series phones but the issue ended up being corrected by the latest phone firmware. We beat up our wireless equipment provider and Nortel (voice and data) and it came down to the phone. I am not sure specifically what was updated in the latest firmware but whatever was changed/added, it did the trick. It is like a new phone…huge difference all around.
Michael McNamara says
That’s very interesting…
We are having issues with some of our older Motorola WS5000 (v2.x software) Wireless LAN Switches with Access Port 200 (802.11a/b) and the Nortel 6140 handsets. I just noticed last week that Nortel released firmware 115.019 for the 6120//6140 handsets so I’ll definitely need to give that a try. It hasn’t been a big issue since we are still using the Nortel 2211 handset and the vast majority of our wireless infrastructure is now Motorola RFS7000/WS5100 (v3.x software with Access Port 300 (802.11a/b/g) which doesn’t exhibit any issues with the 6140 handsets.
Thanks for sharing!
What are thew new changes/fixes with the new version for the 6120 Handsets (115.019)?
We are currently running version115.017.
Richard Springman says
We had been running 097.071 code on our 2211 handsets since a contractor set up our wireless network (Aruba APs, Aruba controller, Nortel 2245) about 3 years ago. While troubleshooting a WLAN VoIP issue recently, we upgraded the 2211 code to 097.072. We noticed right away that the phones would work fine for a while but then would either 1)freeze up 2) start buzzing and not stop or 3)display an error Assert:memory.c Ln 1068. Rebooting the handset would get it back to normal, but the errors would eventually happen again. We rolled the code back to 097.071 and the phones have been steady for the last 24 hours. Have you heard of the 097.072 code causing issues?
PS. Your blog has been a great source of information for me to learn about our own WLAN VoIP. Thanks!
Michael McNamara says
I believe I’m running 097.071 on the few remaining 2211 handsets that we still have around so I can’t really comment. We’ve been migrating to the 6140 handsets for quite sometime now and they have been working really great.
Here’s the MD5SUM of the file on my TFTP server, I believe it’s 097.071 firmware.
Just as a quick note I believe the 2200 series handsets have been EoL and we’re currently having issues getting replacement batteries.
Thanks for the comment and the kind words!
I know the 2211 has been Manufacture Discontinued for a couple of years. I believe the End of Life is August 2012. Thanks for the follow up!