Within the past six months my employer opened a new 1 million square foot distribution facility here in southeastern Pennsylvania. It’s been an exciting challenge for me personally. One of the bigger challenges has been getting the wireless network tuned in since the building opened essentially empty and now almost 4 months later is filled to the gills with product as we prepare for the upcoming holiday season. It’s about 45′ to the ceiling framing joists so I had our integrator mount the APs (Cisco AP 2702e) to 6′ of electrical conduit hanging from the ceiling. This way the APs are a little closer to the intended coverage space yet they are high enough to avoid being hit by the many fork lifts and other vehicles that traverse the distribution center.

Weekly I review the power settings and RF coverage as the building literally fills up with product…which attenuates the signal from the wireless APs little by little. The majority of my WiFi experience is in hospitals, very old buildings and corporate office space where the RF signal only covers about 40-50′ at max and sometimes much smaller (hospitals). The biggest surprise in this facility is the propagation of the RF signal, there’s literally too much RF signal. I’ve had to turn off quite a few AP radios especially in the 2.4Ghz (802.11b/g) band to keep the channel interference to a minimum.

While RF coverage issues are no surprise to anyone working in the industry I have had an interesting time tracking down an idle timeout issue between the Motorola handheld scanners and the IBM AS400 host which runs our warehouse inventory system. The default idle user timeout on the Cisco WLC 5508 is 5 minutes, so I had to changed that parameter to 43200 seconds (12 hours). I also changed the session timeout to 86400 seconds (24 hours). Our employee shifts run 10 hours so these settings looked like they would work – and they do work, except they didn’t work for everyone. In my initial testing I found that I could login from one side of the distribution center, walk to the other side of the distribution center (6 minute walk – it’s that big) and I could pickup my session where I left off without issue even with the handheld going to sleep as I physically moved through the distribution center. I thought the problem was solved and moved onto other issues only to eventually hear that some users were still reporting that they were being “kicked out” – but not everyone.

It was a very specific set of users… those working the receiving dock and the wholesale racking. What was the common denominator? These users would occasionally put down their Motorola handhelds for upwards of 10-15 minutes as they unloaded a truck or prepared to put away a pallet. When they tried to use the handheld it would prompt that their AS400 session had ended and they would need to log in again.

I ran a few tests from the office space in the distribution center and quickly determined that if I was idle longer than 12 minutes I would generally get disconnected from my AS400 session. Was this issue coming from the wireless network or the AS400 host itself? I setup a quick and easy control test. I enabled telnet on one of my Linux jump boxes and established a connection from my test MC9190. After logging into my Linux jumpbox I sat the device on my desk for 25 minutes with no input from me. The Motorola MC9190 went to sleep in about 30 seconds, 25 minutes later I picked up the device, waited for it to power up and connect to the network and found my telnet session still working without issue so this test essentially eliminated the wireless network.

It turns out that the AS400 uses a TCP keepalive to determine if a user is still connected to the system. I’m guessing that since the handheld is sleeping it’s not responding to the keepalive packets so the host system is eventually closing the connection to the handheld. I have some inquires into our AS400 team and IBM to validate my theory.

Cheers!

Update: Sunday October 25, 2015

We were able to update the session keep-alive parameter on the IBM AS400 host. This value defaults to 10 minutes. So every 10 minutes the host will check if the session is still valid by sending a keep-alive packet. If the IBM AS400 host doesn’t get a response to the check it will logoff the session. We increased this value to 20 minutes and that’s helped greatly.

The WS5100 would intermittently come under extreme load for 5-30 minutes, so much load that ultimately the entire wireless network would collapse as the Access Ports started experiencing watchdog resets and would just continually reboot themselves. This problem would come and go throughout the day or night, we could go 12 hours without an issue and then go the next 12 hours with issues every 30 minutes. The problem was affecting both the primary and secondary WS5100 so I eliminated the hardware almost out of the gate. I have first hand experience running v3.3.5.0-002R software on a large number of WS5100s and have never had an issue with that software release so I really didn’t suspect the software. This wireless solution had been in place for more than 18 months without any major issues or problems. The local engineers reported that there had been no changes, no new devices. So what was causing this problem? I immediately suspected an external catalyst but how would I find it?

As with most highly technical problems it wasn’t until I could get my hands on some packet traces and I had time to dissect those packet traces that I could start to fully understand and comprehend what was actually going on.

Topology

A pair of Motorola WS5100 Wireless LAN Switches with 30 AP300 running software release v3.3.5.0-002R in a cluster configuration with one running as primary and the other running as secondary. The network was comprised of a single Cisco Catalyst 4500 with around ten individual Cisco Catalyst 2960S switches at the edge each trunked to the core in a simple hub and spoke design. The entire network was one single flat VLAN. The WS5100s were attached to the Cisco Catalyst 4500 via a single 1Gbs interface, one arm router style. The peak number of wireless devices was around 200, the total number of MAC addresses on the network was around 525 (this includes the wireless devices).

Symptoms

The initial problem report centered around poor wireless performance and sure enough I quickly found 30-40% packet loss while just trying to ping the WS5100. When I finally got logged into the WS5100 I could see that the CPU was running at 100%. The SYSLOG data showed me that the APs were rebooting because of watchdog timeouts. PTRG was showing me that here was a huge traffic surge being received from the WS5100. I quickly realized that the traffic spikes in the graph correspond to events that users were experiencing problems.

Packet Traces

I directed the team to setup a SPAN port to capture the traffic that was flowing between the WS5100 and the Cisco Catalyst 4500 switch. This would provide me a better idea of what was actually on the wire and might provide a clue as to what was transpiring. The team setup Wireshark to continually capture to disk using a 100MB file size and allowing the file to wrap 10 times for a total of 1GB of captured data. The next time the problem occurred I was alerted within 15 minutes by the help desk and users but I found that we missed the start of the event. There was so much traffic Wireshark only had the past 3 minutes available on disk so we had to increase the filesize to 300MB and the number of wrap files to 25 giving us a total capacity of 7.5GB. That configuration would eventually allow me to capture the initial events along with the time needed to get to the laptop and copy the data before it was overwritten. While I waited for the problem to occur I took to setting up SWATCH to alert myself and the team when the problem started so we could quickly gather all the data points during the start of the event.

Using the data from the packet traces we were able to identify and locate two HP desktops that were apparently intermittently flooding the network with ICMPv6 Multicast Listener Reports.

We removed those HP desktops from the network and everything has been stable since.

Analysis

Here’s the current working theory which I believe is fairly accurate. The HP desktops were intermittently flooding the network with ICMPv6 Multicast Listener Reports. Those packets were reaching the WS5100 and because the network at this location is a single flat VLAN the WS5100 needs to bridge those packets over to the wireless network. It does this by encapsulating them in MiNT in a fashion very similar to CAPWAP  or LWAPP. The issue here is the number of packets and the number of access points or access ports. In this case we had 30 APs connected to the WS5100 so let’s do some rough math;

41,000 ICMPv6 Multicast packets * 2 HP desktops = 82,000 packets * 30 APs = 2,460,000 packets

This explains the huge amount of traffic the WS5100 is transmitting. For every ICMPv6 Multicast packet (or broadcast packet for that matter) received by the WS5100, it needs to encapsulate and send a copy of that packet to each and every AP. If there are 30 APs then the WS5100 needs to copy each and every packet 30 times. Now multiply that by the number of ICMPv6 packets that were being received by the WS5100 and you have a recipe for disaster.

A quick search of Google will reveal a number of well documented issues with Intel NICs.

The HP desktops turned out to be HP ProDesk 600 G1s running Windows 7 SP1 with Intel I217-LM NICs driver v12.10.30.5890 with sleep and WoL enabled.

Summary

There were a few lessons learned here;

1. The days of the single flat network are gone. It’s very important to follow best practice when designing and deploying both wired and wireless infrastructures. In this case if the wireless infrastructure had dedicated VLANs both for the wireless client traffic and for the AP traffic this problem would have never impacted the WS5100. It may have impacted the Cisco Catalyst 4500 somewhat but it wouldn’t have caused the complete collapse of the wireless infrastructure. Unfortunately in this case everything was on VLAN 1, wired clients, APs, wireless clients, servers, IP phone systems, routers, everything.
2. The filtering of IPv6 along with Multicast and broadcast traffic from the wireless infrastructure is especially important. I posted back in September 2013 how to filter IPv6, multicast and broadcast packets from a Motorola RFS7000, the same applies to the WS5100. Unless you are leveraging IPv6  in your infrastructure, or have some special multicast applications you should definitely look into filtering this traffic from your wireless network.
3. Validate those desktop and laptop images, especially the NIC drivers and WNIC drivers. In the early days of 802.1x I can remember documenting a long list of driver versions and Microsoft hotfixes required for Microsoft Windows XP (pre SP2) in order to get 802.1x authentication (Zero Wireless Configuration) to work properly.

Conclusion

Wireshark saved this network engineer’s holiday – Thanks!

Cheers!

Note: This is a series of posts made under the Network Engineer in Retail 30 Days of Peak, this is post number 27 of 30. All the posts can be viewed from the 30in30 tag.

I'm troubleshooting a @Motorola WS5100 v3.3.5.0-002R that is being plagued with frequent high CPU utilization and generating 25,000+ pps

— Michael McNamara (@mfMcNamara) December 18, 2014

In the process of trying to understand the problem and come up with a solution I wanted to have better visibility and alerting when the problem actually occurred, I didn’t want to incur the delay that would involve the users calling the help desk and the help desk calling me. Thankfully there is a SYSLOG message recorded when an Access Port experiences a watchdog reset so I had a log message now I needed to find a way to alert on that message.

That’s where I turned to swatch, a handy little utility that will monitor log files for regular expressions and then take whatever action, such as ringing the console or sending an email message is configured. I installed swatch with relative ease thanks to yum and then set out to configure it appropriately.

I created the following configuration file;

#/etc/swatchrc

# swatchrc - define regular expressions and generate alerts when matches are found in logs
#            daemon is started from /etc/cron.d/swatch

# Motorola AP300 - malfunctioning AP ignore events from this device

ignore /00-A0-F8-ZZ-ZZ-ZZ/

# Motorola WS5100 Access Port Adoption Errors Reboot/Watchdog events

#Dec 20 07:53:07 ACME-WLS1 %CC-6-APREADOPTREASON: AP 00-A0-F8-XX-XX-XX readoption reason: ColdBoot/Watchdog
#Dec 20 07:53:25 ACME-WLS1 %CC-6-APREADOPTREASON: AP 00-A0-F8-XX-XX-XX readoption reason: Link failed

# Let's look for the phrase readoption and we'll alert of that text

watchfor /readoption/
        exec "echo '$_' | mail swatch -s 'SWATCH: Motorola WS5100 Adoption Issue' "
        threshold track_by=$6,type=limit,count=1,seconds=60
        echo=red
        bell 5

#end

In the swatch configuration I used the mail aliase of swatch so I edited the /etc/newaliases file to make sure that the entire team would receive the alert;

#
#  Aliases in this file will NOT be expanded in the header from
#  Mail, but WILL be visible over networks or from /bin/mail.
#
#       >>>>>>>>>>      The program "newaliases" must be run after
#       >> NOTE >>      this file is updated for any changes to
#       >>>>>>>>>>      show through to sendmail.
#

# Basic system aliases -- these MUST be present.
mailer-daemon:  postmaster
postmaster:     root

# General redirections for pseudo accounts.
bin:            root
daemon:         root
adm:            root
...
...
...
swatch:         root,mike,john,dan,tom

If the problem is extremely important I’ll usually add the the email SMS text message gateway for my provider. This way I’ll get both an email message and an SMS text message alerting me to the problem.

# Verizon SMS Text Messaging 123456789@vtext.com
# AT&T SMS Text Messaging 123456789@txt.att.net
# T-Mobile SMS Text Messsaging 123456789@tmomail.net
# Sprint SMS Text Messaging 123456789@messaging.sprintpcs.com

I made sure to recompile the aliases file with the newaliases command and then I set off to run swatch in the foreground of my SSH session.

[root@centos /]# swatch -c /etc/swatchrc -p 'tail -f -n 0 /var/log/loc1fac17.log'

*** swatch version 3.2.3 (pid:30643) started at Sat Dec 20 15:54:14 EST 2014

And I waited for the event.

Now I could go about doing some research and due diligence without worrying that I might inadvertently fail to spot the problem.

I’ll let you know how it turned out!

Cheers!

Note: This is a series of posts made under the Network Engineer in Retail 30 Days of Peak, this is post number 26 of 30. All the posts can be viewed from the 30in30 tag.

Here’s an example of WiNG 5.4 forwarding a client to an external captive portal;

"GET /?hs_server=172.16.1.10?Qv=it_qpmjdz=BVQ@bbb_qpmjdz=@dmjfou_njou=532778335@dmjfou_nbd=ED.:C.:D.13.EE.C9@ttj e=VSCBO_HVFTU_XJGJ@bq_nbd=C5.D8.::.33.32.:9 HTTP/1.0" 200 6040 "-" "CaptiveNetworkSupport-305 wispr"

Here’s an example of WiNG 5.5 forwarding a client to an external captive portal;

“GET /?hs_server=172.16.1.10&Qv=it_qpmjdz=BVQ@bbb_qpmjdz=@dmjfou_njou=23:45:9571@dmjfou_nbd=F1.C6.3E.53.:F.:9@ttje=Vscbo _Hvftu_XjGj@bq_nbd=95.35.9E.2:.49.6D HTTP/1.0" 200 6040 "-" "CaptiveNetworkSupport-306.3.1 wispr"

If you look at the URL you’ll notice that the WiNG 5.4 software release utilizes “?” as the variable separator while the WiNG 5.5 software release utilizes “&” as the variable separator.

The Javascript on the captive portal was parsing the URL by splitting it using the “?” value.  I could have just changed the value to a “&” but that would have broken the WS2000s and older RFS 4000s that were using the same webpage and Javascript. As a work around I created an additional path, aup.acme.com/wing55, copied all the files and images into that directory and then editted the Javascript in that folder only. I then reconfigured all the RFS 4000s running WiNG 5.5 to use the new path, example; “http://aup.acme.org/wing55/” and left the remaining devices using “http://aup.acme.org/”.

Cheers!

Reference;
http://www.michaelfmcnamara.com/files/motorola/WiNG5_Captive_Portal_Design_Guide_June_2011.pdf

• MotorolaRFS.RFS7000 (RFS7000)
• MotorolaRFS.RFS6000 (RFS6000)
• MotorolaRFS.RFS4000 (RFS4000)
• MotorolaAP.AP7131 (AP7161)
• MotorolaAP.AP7131 (AP7131)
• MotorolaAP.AP650 (AP650)
• MotorolaAP.AP621 (AP621)
• MotorolaAP.AP6521 (AP6521)
• MotorolaAP.AP6532 (AP6532)
• MotorolaAP.AP6511 (AP6511)

The APs will try to associate via a Layer 2 broadcast with a controller, if they fail to adopt via Layer 2 they will issue a a DHCP request with the vendor class IDs listed above.

This is really helpful in your IPAM or DHCP server, you can define specific pools based on the vendor class to return very specific DHCP options. In this case you would probably want to return DHCP option 189 (string) with the IP address of the wireless LAN switch. You can include multiple IPs separated by commas.

Cheers!

Update: Sunday June 2, 2013 Motorola has responded with the following analysis and explanation of the problem.

The Association Request from the Samsung Galaxy S4 phone has the RRM (Radio Resource Management/802.11k) capability element missing, but the Capabilities Info Bitmap in the Association Request says that the client can do RRM, so there is a mismatch and we deny association in WiNG4.x. In WiNG5.x, the RRM implementation is different and we don’t enforce such a strict check to avoid situations such as these where clients might not be following the 802.11k specification properly. In WiNG 3.x, there is no support for RRM, so there are no such checks enforced.

Thanks to Motorola for providing the quick analysis and explanation. Just to summarize the problem is not present in WiNG 3.x or WiNG 5.x but is definitely present in WiNG 4.x. As for a workaround or fix I’m still waiting to hear if Motorola will issue a patch (software release) or if they will leave it to Samsung and Google to resolve.

Update: Thursday May 30, 2013 It seems that the problem is not evident when the Samsung Galaxy S4 associates with a Motorola WS5100 (v3.3.2.0-010Ri) with AP300s as access ports.

It would seem that the recently released Samsung Galaxy S4 is having difficulty connecting to our public wireless network which is provided by a pair of Motorola RFS 7000 Wireless LAN Switches (v4.4.2.0-001R) with about 24 AP650s (v2.2-1592R). While I’ve personally observed this problem in our office, I’ve also received similar reports from users in our hospitals which are running RFS7000s with either AP300s or AP650s for Access Ports/Points.

Our public wireless network has no authentication or encryption, however, the Samsung Galaxy S4 will display “Authentication error occurred” when it tries to connect. I performed a quick wired packet trace on the captive portal server and found no frames with the associated MAC address of the Galaxy S4 so I setup WireShark with 3 AirPcap adapters, one for each channel in the 802.11b/g  2.4Ghz range, to perform a wireless packet trace.

I was able to observe the Galaxy S4 making repeating probe requests and association requests but every association attempt appears to fail with an Unspecified error. As a reference I also captured my Motorola Droid 3 connecting to our public network for comparison.

WireShark AirPcap Captures

• First run of Samsung Galaxy S4 – failed to connect
• Second run of Samsung Galaxy S4 – failed to connect
• Working run of Motorola Droid 3 – working

Looking at the wireless packet trace I can see where the Galaxy S4 is failing to associate to the network. In frame 341 we can see “Unspecified failure” in the Association Response from the Access Port. I’m not an expert here but I’m going to guess that there’s something in the Association Request that is causing the wireless infrastructure to choke on the response.

Looking at the last wireless packet trace of the working Motorola Droid 3 we can see that it quickly probes, associates and makes a DHCP request without any problems or issues.

My Thoughts

As I’ve mentioned before I’m no expert here but I can see quite a few additional tags in the Association Request from the Samsung Galaxy S4. I’m going to guess that it’s one of these tags that is causing the wireless infrastructure to choke. Looking at the screenshot below you can see all the tags.

I’m hoping some wireless experts can step up here, or perhaps Motorola with an explanation and workaround/fix?

Cheers!

If I removed the security settings on the WLAN everything appeared to work fine. I even went so far as to enable the wireless capability of the Verizon FiOS Actiontec router that I have installed with my Verizon FiOS Internet/TV and was successful in connecting the Nintendo DSi to that WLAN with WEP (64bit) security. I should have setup a wireless packet trace and perhaps even a packet trace on the WS2000 but I didn’t feel like dealing with all that work, I felt like a user, I just wanted it to work – why can’t it just work?

I know the WS2000 configuration is correct and working because I have an HP EliteBook 8440p and Motorola Droid connecting to the same WLAN without issues. So it would seem that there is some issue between the Nintendo DSi and the Motorola WS2000 if you try to use WPA2-PSK/AES-CCMP security (I wonder if the same issue is evident on the WS5100 and/or RFS7000?). I believe there is an update available for the WS2000 although I didn’t have the time to apply that update this evening. I may apply that update in the future and update this post accordingly. For now it seems I’ll need to stick with the Verizon FiOS Actiontec router wireless for the Nintendo DSi.

Cheers!

In any event if you are ordering/deploying any new AP5131 or AP300s you’ll need to be mindful of this change and ensure that you are running the appropriate software releases and/or you have applied the specific patches.

You can find the release notes concerning the AP5131 right here.

You can find the release notes concerning the AP300 right here.

Cheers!

Motorola has released software v3.3.1 for the WS5100 and v1.3.1 for the RFS7000 Wireless LAN Switches. You can find the release notes for the 3.3.1 (WS5100) software release here. And you can find the release notes for the 1.3.1 (RFS7000) software release here. We’ve been running v1.3 on the RFS7000 for the past few months now with only a few small problems. We hope to start testing the Smart RF feature set that was released in the 1.3 Wi-NG software base very soon. We’re also eager to start testing the AP7131 802.11n Access Port in a few specific locations.

Here’s a quick excerpt from Motorola on v1.3.1 for the RFS7000;

RFS7000 v1.3.1 has the following feature focus: Voice, Security and Resiliency

Voice: Enhancements provide comprehensive WMM Admission control, enabling not only superior voice quality but also optimizations with respect to network usage for voice.

Security: Enhances the built-in IDS capabilities for Ad-Hoc Network Detection and .11n Rogue detection. Provides built-in IPS capabilities via Rogue AP containment for the wireless network.

Resiliency: SMART RF Management that enables the WLAN to automatically and intelligently adapt to changes in the RF environment to eliminate unforseen gaps in coverage.This technology provides dynamic network optimization to ensure user quality of experience at all times by automatic adjustments to channel and power (on detection of RF interference or loss of RF coverage/neighbor recovery).

All the above enable the wireless enterprise by making it easy to deploy, securely and with built-in resiliency and support for voice.
For the Adaptive AP:
• Adaptive AP7131 802.11 a/b/g/n Support ( v3.1.1 )
• Rogue AP detection
• Mesh statistics
• WLAN statistics
• Configurable IPS Sensor on the AP5131( D SKU) in Adaptive mode(ADP-5131 v2.2.1 image)
With the AP300:
• Dynamic Load balancing of AP300s after a primary reverts in a cluster
• Email Notification for critical alarms
• LDAP enhancements
• Cluster GUI for WLANS and APs visualization
• Securing Layer 3 AP and Wireless Switch protocol – Secure WiSPe
• MU Naming
System Enhancements:
• IP v6 Client Support

Cheers!

RFS7000*>
RFS7000*>enable
RFS7000*#service clear ?
all          Remove all core, dump and panic files
aplogs       Remove all local ap log files (does not clear them off the AP)
clitree      Remove clitree.html (created by the save-cli command)
cores        Remove all core files
dumps        Remove all dump files
panics       Remove all kernel panic files
securitymgr  Securitymgr parameters
RFS7000*#>service clear all
RFS7000#

Cheers!

It was a challenge to understand all the different heartbeats, timeouts and protocols that were in play between the handset and the Nortel 2245 wireless gateway and ultimately the Nortel Succession Signaling Server. With any Nortel IP phone running a UNIStim protocol there is a watchdog timer on the phone that counts down from 200 seconds. The watchdog timer must be reset by a watchdog reset (heartbeat) message that gets sent out from the Nortel Succession Signaling Server. This watchdog reset gets sent every 30 seconds. If a handset, remember now any Nortel IP handset that is running a UNIStim protocol such as the i2002, i2004, 1120e, 1140e, 1150e, 2210, 2211, 6120 and 6140 misses too many of these heartbeats the phone will reset itself usually displaying the message “watchdog timeout” indicating that the watchdog timer has reached zero and the phone is attempting to recover from the problem by resetting itself. With the Nortel 2210, 2211, 6120 and 6140 you also have the SVP heartbeats and timeouts to worry about.

If you have some IP phones that are generating “watchdog timeout” message your probably loosing packets somewhere in your network. With that said I would advise anyone with such a problem to immediately contact their voice reseller and make sure their Succession Call Server and Signaling Server have the latest and greatest DEP (patches) list. Once that’s complete you’ll need to go about the task of isolating the possible locations where you could be dropping packets. If it’s a wired IP phone then the problem is much easier to troubleshoot and isolate. If it’s a wireless phone then you’ll have a few extra steps. You’ll obviously need to make sure that you have QoS (DiffServ) up and working within your environment and you’ll need to make sure that you have SVP support enabled on your wireless infrastructure. SpectraLink (recently acquired by Polycom) actually has a library of documents to help customers configure their wireless infrastructure properly to support the SpectraLink handsets.

Cheers!

Correction: August 19, 2008
The watch dog interval is actually 200 seconds long and not 120 seconds as originally posted.

Update: August 24, 2008
It would seem that this article has generated a lot of interest including several inquiries by Nortel. So I thought I would try to add some additional explanation to help more clearly describe the problems and experiences I’ve had the Nortel 2211 and 2210 wireless handsets. I won’t rewrite the original because I don’t think there is anything wrong with it, other than perhaps missing some attention to the specific details.

The Motorola WS5100 v3.x and RFS7000 v1.1 was technically broken for anyone using the Nortel 2211/2210/6120/6140 wireless handsets. The phones would often reset while idle, because of a buffering issue on the Motorola AP300 access port. These problems have been resolved (as far as my testing indicates) in the Motorola WS5100 v3.2 and RFS7000 v1.2 software release. Through our troubleshooting of this problem we learned a great deal about the Spectralink Voice Priority protocol and the UNIStim protocol. In short the Nortel wireless handsets will go into PSP (Power Save Polling) for approximately 1.5 seconds, during that time the wireless handset turns off it’s radio to help save power and preserve the battery life. The problem occurs while the phone is idle because of the PSP mode, this is why no problems are ever reported while the phone is off-hook and actively being used. While the wireless handset is in PSP mode the wireless network is responsible for buffering any packets that are sent to the handset. The SVP protocol and UNIStim protocol can generate a lot of packets causing the wireless network to discard some packets while the phone is in PSP mode. These discarded packets can, depending entirely on the timing, cause the phone to either reset or the phone to be unregistered from the Succession Signaling server.

I’ve been asked by quite a few people what can be done to help alleviate any potential issues?

• The wireless infrastructure should be configured to support the SVP protocol
• QoS (DiffServ) should be set to “Trusted” on every Ethernet switch port that will be used to connect the different equipment (Succession Signaling Server, Succession Voice Gateway Media Card, 2245, wireless infrastructure)
• Design the wireless infrastructure so there is at least -60 dB of signal available and no more than 7 wireless handsets connected to a single access point/access port.

With all that said Nortel has literally just released v97.072 software for the Nortel 2211/2210 wireless handsets. While the release notes don’t seem to indicate any changes that are specific to “watchdog” issues it might be worth giving it a shot.

Cheers!

Update: Friday September 12, 2008
I’ve placed a copy of the Nortel document WLAN IP Telephony Installation and Commissioning (v3.3) on my website. This document should be a great help to many folks that are having issues with Nortel 22×0 and 61×0 wireless handsets.

You’ll be using the CLI interface to perform the upgrade; there will be no need for the web Java GUI until after the upgrade is complete.

[root@madmax ~]# telnet sw16r-wireless.tlh.acme.org
Trying 10.115.255.253...
Connected to sw16r-wireless.tlh.acme.org (10.115.255.253).
Escape character is '^]'.
user name: cli

When prompted for the “user name” use “cli”. When prompted for the “userid” use the default of “admin” and “symbol” as the password.

Symbol Wireless Switch WS 5000 Series.
Please enter your username and password to access the Command Line Interface.

userid: admin
password: *********

Retrieving user and system information...

Setting user permissions flags..
Checking KDC access permissions...

Welcome...

Creating the Event list...
System information...

System Name                  : sw16r-wireless
Description                  : WS5000 Wireless Network
Switch Location              : Data Center
Software Ver.                : 1.4.3.0-012R
Licensed to                  : Symbol Technologies
Copyright                    : Copyright (c) 2000-2005.  All rights reserved.
Serial Number                : 00A0F865B362
Number of Licenses           : 0
Max Access Ports             : 30
Max Mobile Clients           : 4096
Active Switch Policy         : Wireless Switch Policy
Emergency Switch Policy      : Not defined
Switch Uptime                : 35d:23h:41m
# of Unassigned Access Ports : 0

sw16r-wireless>

It’s advised to start out by backing up the switch configuration and then uploading that configuration to the TFTP server on the network. You’ll first need to delete the existing configuration file. (If the switch is a standby switch there is no need to backup the configuration file).

sw16r-wireless> del sw16-wireless.cfg
Removing sw16-wireless.cfg.... done.

sw16r-wireless> save configuration sw16-wireless.cfg
Saving running configuration in: sw16-wireless.cfg

Saving wireless network management configuration...
Configuration saved successfully.

sw16r-wireless> copy sw16-wireless.cfg tftp://10.101.20.1/sw16-wireless-tlh.cfg

Copying 'sw16-wireless-tlh.cfg' from Switch to tftp://10.101.20.1...
File: sw16-wireless-tlh.cfg copied successfully to 10.101.20.1

Once you’ve backed up the switch configuration you need to make room for the new image. Delete all the files from the flash memory. You can use the “dir” command and “del” command.

sw16r-wireless> dir
Date & Time        Bytes  File Name

Mar 29  2005        15480  WS5000Defaults_v1.4.1.0-014R.cfg
Jan 24  10:46    19591051  WS5000_v1.4.3.0-012R.sys.img
Jan 24  10:48       16138  WS5K_v1.4.1.0-014R-Upg.cfg
Oct  3  2005         6517  cmd_template.sym
Oct  3  07:22       17345  sw16-wireless-tlh.cfg

sw16r-wireless> del WS5000Defaults_v1.4.1.0-014R.cfg
Removing WS5000Defaults_v1.4.1.0-014R.cfg.... done.
sw16r-wireless> del WS5000_v1.4.3.0-012R.sys.img
Removing WS5000_v1.4.3.0-012R.sys.img.... done.
sw16r-wireless> del WS5K_v1.4.1.0-014R-Upg.cfg
Removing WS5K_v1.4.1.0-014R-Upg.cfg.... done.
sw16r-wireless> del cmd_template.sym
Removing cmd_template.sym.... done.
sw16r-wireless> del sw16-wireless-tlh.cfg
Removing sw16-wireless-tlh.cfg.... done.

Now you can go ahead and download the new system image and accompanying files via FTP. I’ve already placed the system image on the FTP server. The following files will need to be downloaded from the FTP server (10.101.20.1); WS5000_v2.1.0.0-029R.sys.kdi, dominfo, PreUpgradeScript, WS5k_domfix.cfg. You can confirm that the file gets copied down by listing the directory contents using “dir”.

sw16r-wireless> copy ftp system -u mcnamm
Enter the file name to be copied from FTP server : PreUpgradeScript
IP address of the FTP server : 10.101.20.1
Enter the user password : **********

Copying 'PreUpgradeScript' from ftp://10.101.20.1 to Switch...
Data connection mode : BINARY (Connecting as 'mcnamm')

Status : Transfer completed successfully
19633 bytes received in 0.0098 seconds (2e+03 Kbytes/s)
/bin/dedos: line 69: syntax error near unexpected token `dir'
/bin/dedos: line 69: `dedos -R

sw16r-wireless> copy ftp system -u mcnamm
Enter the file name to be copied from FTP server : dominfo
IP address of the FTP server : 10.101.20.1
Enter the user password : **********

Copying 'dominfo' from ftp://10.101.20.1 to Switch...
Data connection mode : BINARY (Connecting as 'mcnamm')

Status : Transfer completed successfully
48346 bytes received in 0.015 seconds (3.2e+03 Kbytes/s)

sw16r-wireless> copy ftp system -u mcnamm
Enter the file name to be copied from FTP server : WS5k_domfix.cfg
IP address of the FTP server : 10.101.20.1
Enter the user password : **********

Copying 'WS5k_domfix.cfg' from ftp://10.101.20.1 to Switch...
Data connection mode : BINARY (Connecting as 'mcnamm')

Status : Transfer completed successfully
1410387 bytes received in 0.15 seconds (9.5e+03 Kbytes/s)
Verifying configuration file...
Valid configuration file. Completing verification.

sw16r-wireless> copy ftp system -u mcnamm
Enter the file name to be copied from FTP server : WS5000_v2.1.0.0-029R.sys.kdi
IP address of the FTP server : 10.101.20.1
Enter the user password : **********

Copying 'WS5000_v2.1.0.0-029R.sys.kdi' from ftp://10.101.20.1 to Switch...
Data connection mode : BINARY (Connecting as 'mcnamm')

Status : Transfer completed successfully
39661568 bytes received in 22 seconds (1.8e+03 Kbytes/s)

sw16r-wireless> dir
Date & Time        Bytes  File Name

Oct  3  07:28       19633  PreUpgradeScript
Oct  3  07:29    39661568  WS5000_v2.1.0.0-029R.sys.kdi
Oct  3  07:28     1410387  WS5k_domfix.cfg
Oct  3  07:28       48346  dominfo

sw16r-wireless>

The next step is to execute the PreUpgradeScript and check if there is adequate space for the upgrade. You’ll need to enter “service mode” to execute the following commands. You can enter “service mode” by entering the command “service”. The password may either be “password” or the switch admin password.

sw16r-wireless> service
Enter CLI Service Mode password: ********
Enabling CLI Service Mode commands...... done.

SM-sw16r-wireless> launch -c chmod +x /image/PreUpgradeScript

SM-sw16r-wireless> launch -c /image/PreUpgradeScript freemem
PreUpgradeScript : freemem - computing Free memory
DOM firmware upgrade will NOT be performed
Finding out the Free Space Needed ... !!
Total Free Space on the System: 148 (in MB)
OK. Required space to do the upgrade exists .. !!

If you receive the “OK” you can go ahead with the upgrade. It may be necessary (with Wireless LAN Switch 5000s) to run the “PreUpgradeScript freemem” prior to downloading the WS5000_v2.1.0.sys.kdi image. The 5000 switches only have 128Mb of flash space available.

SM-sw16r-wireless> launch -c /image/PreUpgradeScript upgrade
PreUpgradeScript : upgrade - upgrading the system
Deciding on DOM firmware upgrade, based on switch platform
This is a butterfly 1.4.x series switch
This is WS5100 switch, no need for firmware upgrade
Verifying checksum for : dominfo
Checksum verification for dominfo : passed
Showing details of DOM

Model Number______________________: Kouwell DOM
Serial Number_____________________: HyFlash     00004020
Controller Revision Number________: 14/05/02

Able to do Double Word Transfer___: No
Controller buffer size (bytes)____: 512
Transfer Speed____________________: > 10 Mbit/sec
Drive Type________________________: Removable
IORDY Supported___________________: No
Can IORDY be disabled by device___: No
LBA Mode supported________________: Yes
DMA Supported_____________________: No
Number of ECC bytes transferred___: 4
Number of sectors per interrupt___: 1

Number of Cylinders_______________: 980
Number of Heads___________________: 16
Number of Sectors per Track_______: 32

Enter the Image Name: WS5000_v2.1.0.0-029R.sys.kdi
Verifying Image Checksum
Image Checksum Verification Passed
Saving the Configuration before upgrading
Saving wireless network management configuration...
Configuration saved successfully.
Creating the configuration tar
tar: Removing leading / from absolute path names in the archive.
image/upgrade.cfg
Copying the image
Rebooting the system
Shutting down snmpd agent.....done.
Shutting down apache server...done.
Shutting down cell controller.......done.
Shutting down database main thread...done.
Rebooting the switch...
Connection closed by foreign host.

Now you’ll need to wait.; it should take between 5 and 10 minutes for the switch to upgrade and reboot. After the switch has rebooted you can re-establish your telnet session;

[root@linux ~]# telnet sw16r-wireless.tlh.acme.org
Trying 10.115.255.253...
Connected to sw16r-wireless.tlh.acme.org (10.115.255.253).
Escape character is '^]'.
=========== WS5000 Switch ===========

Copyright(c) Symbol Technologies, Inc. 2005.
All rights reserved.

user name: cli

Symbol Wireless Switch WS 5000 Series.
Please enter your username and password to access the Command Line Interface.

userid: admin
password: *********

Retrieving user and system information...

Setting user permissions flags..
Checking KDC access permissions...

Welcome...

Creating the Event list...
System information...

System Name                  : sw16r-wireless
Description                  : WS5000 Wireless Network
Switch Location              : Data Center
Software Ver.                : 2.1.0.0-029R
Licensed to                  : Symbol Technologies
Copyright                    : Copyright (c) 2000-2005.  All rights reserved.
Serial Number                : 00A0F865B362
Number of Licenses           : 0
Max Access Ports             : 30
Max Mobile Clients           : 4096
MU Idle Timeout value        : 1800  seconds
Active Switch Policy         : Wireless Switch Policy
Emergency Switch Policy      : Not defined
Switch Uptime                : 00d:00h:03m
Global RF stats              : Disabled
# of Unassigned Access Ports : 0
CLI AutoInstall Status       : Enabled

sw16r-wireless> copy tftp system
Enter the file name to be copied from TFTP server : WS5000_v2.1.1.0-006R.sys.img
IP address of the TFTP server : 10.101.20.1

Copying 'WS5000_v2.1.1.0-006R.sys.img' from tftp://10.101.20.1 to Switch...
File: WS5000_v2.1.1.0-006R.sys.img copied successfully from 10.101.20.1
Verifying imagefile...
Valid imagefile. Completing verification.

sw16r-wireless> restore system WS5000_v2.1.1.0-006R.sys.img
This command will reset the system and boot up with the new restored image.
Do you want to continue (yes/no)  : yes

Restoring system image and configuration from WS5000_v2.1.1.0-006R.sys.img
It might take a few minutes.......

Saving wireless network management configuration...
Configuration saved successfully.
Stopping Postgres database.. done
Creating Default Configuration file for 2.1.1.0-006R..

Rebooting the switch...

Shutting down dhcp daemon.. done
Shutting down apache server in the SSL mode...done.
Cell controller not running.
Shutting down Postgres....done.
Connection closed by foreign host.

You’re all done.

The only issue I’ve discovered is that you need to re-configure the SNMP community string and TIMEZONE on any upgraded switch.

Enjoy.

You should connect to the console port a serial cable (null) with 19200,8,N,1.

The example below will configure Ethernet 2 as a trunk port with the management interface in VLAN 200 (10.107.255.199/24) and the default gateway as 10.107.255.1. The order of the commands is very important when you start to trunk the interface.

Please press Enter to activate this console.
WS5100 release 3.0.3.0-003R
Login as 'cli' to access CLI.
WS5100 login: cli

User Access Verification

Username: admin
Password: *********
Welcome to CLI

WS5100>
WS5100> enable
WS5100# configure terminal

WS5100(config)# interface eth2

WS5100(config-if)# switchport mode trunk
WS5100(config-if)# switchport trunk native vlan 200
WS5100(config-if)# switchport trunk native tagged
WS5100(config-if)# switchport trunk allowed vlan none
WS5100(config-if)# switchport trunk allowed vlan add 200
WS5100(config-if)# exit

WS5100(config)#interface vlan 200
WS5100(config-if)# ip address 10.107.255.199/24
WS5100(config-if)# management
WS5100(config-if)# exit

WS5100(config)# interface vlan 1
WS5100(config-if)# no ip address
WS5100(config-if)# shutdown
WS5100(config-if)# exit
WS5100(config)# ip default-gateway 10.107.255.1
WS5100(config)# end
WS5100# write memory

Once you’ve complete those steps you should be able to ping the device. At that point you can connect to the web based console to complete the configuration.

https://10.107.255.199

You should of course substitute the IP addresses above with your own addresses.

Cheers!