We had an issue a few months back with a number of Elo Touch all-in-one systems. These devices had been installed and working for almost three years and then literally overnight they started having issues connecting to our wireless infrastructure – all at the same time. Oddly enough the issue was only impacting the Elo devices, we had numerous other devices including Lenovo laptops, macOS laptops, Apple iPhones, Zebra TC20/TC21 Handhelds (Android), Zoom Conference TVs (Apple Mac Mini) all working without issues or problems. The initial troubleshooting didn’t turn up anything simple, there were no locked out accounts or other RADIUS 802.1X authentication issues. We just didn’t see the devices in question even trying to associate to any of the APs so we were initially stumped. While we worked to get an engineer onsite we performed the obligatory rolling reboot of the Cisco WLC 5520s (primary and standby) along with the Cisco AP 4800s (they had an uptime of just over 645 days) just to check that box for lack of any other direction at that time.

What was the issue?

In this specific facility we only use the 5Ghz band for our production networks, 2.4Ghz is setup for the guest network. In the end we determined (still waiting on confirmation) that the devices in question don’t appear to support all the 802.11a 5Ghz wireless channels. We found the following reference on several Internet websites.

Elo devices cannot operate on 5G wireless networks utilizing 5.250 to 5.350 GHz OR 5.470 to 5.725 GHz.

I didn’t know the frequencies off the top of my head so I had to look them up… thanks to the folks at Wireless LAN Professionals for the chart below. That potentially removes channels 52-64 and channels 100-144 from being used, only leaving channels 36-48 and I would have to guess the device likely doesn’t support the UNII-3 band and channels 149-165 so that’s super restrictive.

In a large fulfillment center it’s usually feast or famine, too much RF signal or not enough RF signal and it takes a lot of work to find that happy medium.

What happened?

It would appear that Dynamic Channel Assignment (DCA) on the Cisco WLC 5520 changed an AP from channel 48 to channel 136 the morning the issue started, found the log entry, and that was the only AP in the physical area around the clients that was using any of the channels between 36 and 48. In short the Elo devices were blind to the wireless access points around them because they were on channels that the devices didn’t support. This was later confirmed by performing some remote wireless packet traces from some one of the Cisco 4800 APs in sniffer mode. We captured numerous packet traces across numerous 5Ghz channels but we were unable to see any of the Elo devices communicating in any channel other than 36-48. We were looking for active probe requests in the wireless packet traces which is not fool proof as the client can still listen passively. We manually set the AP back to channel 48 and the devices immediately started working. We’ve temporarily disabled TPC and DCA while we try to validate what channels the device supports.

The Elo vendor reps we contacted claimed that the devices support all the “standard” 5Ghz channels but from the evidence we collected that doesn’t appear to be the case. I hope to be able to get my hands on one of these devices in the coming weeks to try and validate my suspicions.

I still need to confirm but this is really the only explanation that fits the available evidence.

Anyone else ever have such an odd problem?

Cheers!

Update: July 2022

I was able to get my hands on ELO and was able to verify that it could in fact communicate in the UNII-2a bands, so I’m not sure what to make of this issue with that new technical tidbit.

It turns out there are multiple models of the Lenovo ThinkPad T14, one with an Intel wireless NIC and one with a Realtek wireless NIC. We quickly discovered that the model with a Realtek RTL8852AE WiFi 6 802.11ax PCIe adapter was having a lot of issues staying connected to a number of different Cisco Wireless LAN Controllers in different physical locations. The symptom displayed to the user as an inability to pull a DHCP address, even though the device showed it was connected to the SSID. In the end it turns out that a driver released on August 10, 2021 (6001.0.10.334) that apparently fixes an issue when clients are using a Cisco wireless infrastructure. Unfortunately there’s no mention of what exactly the issue was in the release notes.

You can find the updated driver and release notes at the following link;

https://pcsupport.lenovo.com/us/en/products/laptops-and-netbooks/thinkpad-t-series-laptops/thinkpad-t14s-type-20uh-20uj/downloads/driver-list/component?name=Networking%3A%20Wireless%20LAN

I’ve been seeing a lot of issues as we move to WiFi 6 access points – currently rolling out Juniper MIST AP43s. And in the vast majority of these cases older drivers are the problem. A quick upgrade to the latest and greatest driver is solving the majority of issues. So if you are having issues with the WiFi 6 based access point or client, I would strongly suggest you update your driver before you fire up WireShark.

Cheers!

I had another opportunity to try my hand at a point to point wireless solution albeit somewhat reluctantly. In this case the distance that we needed to span was about 1,500 ft. The construction costs to pull fiber into this building was in excess of $50,000 so I reluctantly starting looking at wireless solutions. Ultimately I landed on the Ubiquiti NanoBeam AC Gen2 (NBE-5AC-GEN2-US) solution. It was relatively inexpensive and wouldn’t cost much to actually install and see if it would work. That was 18 months ago… and that sucker hasn’t dropped a packet since [knock knock].

In real world iPerf testing we get about 350 Mbps through the link which is pretty awesome considering the solution was essentially under $1000 with the surge arresters and installation.

I can’t say that everyone will have this level of success… but seeing how relatively inexpensive these devices are it seems well worth the gamble given that you might pay $1,500/monthly for a 500Mbps Ethernet circuit… that’s $18,000/yearly. If it works the solution will essentially pay for itself in the first month of operation.

Now don’t go all crazy… you’ll need line of site between the two locations and growing trees can present interesting problems. And if a new building or structure pops up between your wireless endpoints, well you’ll be out of luck.

You might be asking so what did I do to resolve the issues with the infrared cameras 10 years ago? We were able to contract with Sunesys, now Crown Castle to lease 4 fiber strands between the two buildings. The local electrical utility wanted $40,000 to replace two telephone polls when I tried to pull the fiber myself so we went with Sunesys who had the fiber installed and running in 2 weeks – no new telephone polls required.

Are you running any point to point wireless solutions?

Cheers!

In this specific case I had a pretty good idea why they couldn’t play Minecraft together over our home network and it wasn’t something that they were likely to figure out searching through Google Search or watching YouTube videos.

I use my personal home network as a testbed of sorts – and wireless is one technology where you need to-do a lot of testing before pushing it into production. I this case I had replaced my trusty Aruba Instant AP-215s with AP-315s in my home and was testing the newer APs and the 8.3.0.0 software. I’ve got a little of everything on my home wireless network, Windows 10, Windows 8.1, MacOS, iOS, Ubuntu Linux, Lenovo, Acer, Apple iPhone, Apple iPod, Roku, Samsung, Amazon Fire, Motorola, Nest Cameras, and various other IOT devices.

The Aruba Instant APs are configured by default to filter all broadcast packets except ARP requests. This prevents the broadcast that Minecraft emits from reaching other players connected to the same home network.

Once I changed the setting to none and applied the changes the girls were able to play Minecraft together via our home network.

The Aruba IAP-315s are 802.11ac Wave 2 access points and support 5GHz 4×4 MIMO (1,733Mbps max rate) and 2.4GHz 2×2 MIMO (300Mbps max rate) radios. I’m pretty happy so far with the testing… no disconnect issues or performance problems.

Cheers!

The Aruba notification cited the following two articles;

• Aruba products contain compromised HTTPS certificate
• Cryptographic Key Reuse Remains Widespread In Embedded Products

In summary, a third party named SEC Consult posted a large number of private keys to GitHub including the private key for securelogin.arubanetworks.com which they had extracted from an Alcatel-Lucent OmniAccess Wireless Access Point. It just happens that Aruba is the OEM for the Alcatel-Lucent OmniAccess product line. With the private key public, it’s fairly easy for anyone to perform a man-in-the-middle attack and eavesdrop on supposedly secure traffic. Shortly afterwards GeoTrust revoked the certificate which will now cause browsers to throw an error when using the certificate for guest registration or captive portal.

This will only happen for browsers which have recently updated their CRL list. If we check the CRL we’ll find the certificate with serial number 01DA52.

[root@centos ~]# wget http://gtssldv-crl.geotrust.com/crls/gtssldv.crl
--2016-09-28 20:15:01--  http://gtssldv-crl.geotrust.com/crls/gtssldv.crl
Resolving gtssldv-crl.geotrust.com... 2600:1400:a:18f::1abd, 2600:1400:a:18b::1abd, 23.50.69.163
Connecting to gtssldv-crl.geotrust.com|2600:1400:a:18f::1abd|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5150 (5.0K) [application/pkix-crl]
Saving to: `gtssldv.crl'

100%[======================================>] 5,150       --.-K/s   in 0s

2016-09-28 20:15:01 (189 MB/s) - `gtssldv.crl' saved [5150/5150]

[root@centos ~]# openssl crl -inform DER -text -noout -in gtssldv.crl
Certificate Revocation List (CRL):
        Version 2 (0x1)
    Signature Algorithm: sha1WithRSAEncryption
        Issuer: /C=US/O=GeoTrust Inc./OU=Domain Validated SSL/CN=GeoTrust DV SSL CA
        Last Update: Sep 28 23:43:00 2016 GMT
        Next Update: Oct  8 23:43:00 2016 GMT
        CRL extensions:
            X509v3 Authority Key Identifier:
                keyid:8C:F4:D9:93:0A:47:BC:00:A0:4A:CE:4B:75:6E:A0:B6:B0:B2:7E:FC

            X509v3 CRL Number:
                164147
Revoked Certificates:
...
    Serial Number: 01DA52
        Revocation Date: Sep  7 17:04:44 2016 GMT
...
    Signature Algorithm: sha1WithRSAEncryption
         58:20:83:3f:5c:cf:31:0f:69:9b:e5:af:dd:ca:2f:b9:e1:42:
         b0:bb:68:f3:b8:e8:8c:e2:b1:17:0b:bf:5a:d0:a2:65:3c:1f:
         18:bf:93:3a:55:ea:25:c9:da:ba:3b:ed:b6:c7:67:ae:33:b2:
         34:40:27:b9:5c:76:8a:3a:8d:4e:8a:9c:d3:61:a7:ed:01:a0:
         94:fe:ab:7d:96:79:cb:4e:a5:bb:2e:fb:3c:96:52:3e:ae:e9:
         9a:d9:0b:59:3a:c0:f6:45:10:3e:f2:b4:d1:79:63:0b:41:ed:
         d4:70:d6:26:67:1d:ab:7a:6e:c8:8d:6a:0e:df:17:ff:b3:e6:
         a7:dd:6c:5f:fe:41:79:0c:09:51:cd:6d:7d:33:2a:0b:48:a2:
         5c:5c:5c:06:f3:6b:d6:b9:af:14:34:3d:7d:b9:85:36:32:38:
         c1:70:50:9d:13:01:ab:b1:de:78:0d:10:24:f2:56:3d:d2:77:
         93:d1:70:b8:47:78:a6:22:aa:6a:95:af:49:cb:bc:f6:1e:dd:
         1b:0e:bd:10:54:09:39:93:4a:10:a3:1f:76:99:12:47:35:0d:
         37:95:6e:fc:a1:b5:e1:f0:d4:f7:96:3e:52:44:e4:69:d9:64:
         37:ec:15:50:fd:e8:f3:06:3b:ee:c1:6a:ee:0a:01:fc:fc:e0:
         7e:92:d8:0c

We had to replace the certificate, but how do you replace the certificate on almost 600+ Aruba Instant AP Virtual Controllers? That’s where we looked at utilizing Airwave Management Platform (AMP), and the process appeared to be pretty straight forward although like almost everything with technology we quickly ran into problems. We started by getting a new public certificate and combined the certificate, intermediate certificate and root certificate all into a single file. We uploaded that file to AMP and then went through the multiple groups within Airwave to enable the new certificate.  And we quickly thought, that was too easy and it was.  While a number of groups worked and the VCs in those groups were getting the new certificate, a large number of groups were not working and those VCs were not getting the new certificate. We turned to Auba TAC should guided us through the following steps 1) upgrade of AMP from 8.0.9.2 to 8.0.10, 2) upgrade of the IAPs from 6.3.1.2-4.0.0.4  to 6.4.2.6-4.1.3.1, 3) second upgrade of AMP from 8.0.10 to 8.2.2. We finally discovered that the following was missing from the template; %captive_portal_cert_checksum%. Without this line in the configuration template AMP will not push down the certificate to the IAPs.

Cheers!

References:

ArubaOS Default Certificate Revocation FAQ – Instant

Image Credit: Eliseeva Ekaterina

Within the past six months my employer opened a new 1 million square foot distribution facility here in southeastern Pennsylvania. It’s been an exciting challenge for me personally. One of the bigger challenges has been getting the wireless network tuned in since the building opened essentially empty and now almost 4 months later is filled to the gills with product as we prepare for the upcoming holiday season. It’s about 45′ to the ceiling framing joists so I had our integrator mount the APs (Cisco AP 2702e) to 6′ of electrical conduit hanging from the ceiling. This way the APs are a little closer to the intended coverage space yet they are high enough to avoid being hit by the many fork lifts and other vehicles that traverse the distribution center.

Weekly I review the power settings and RF coverage as the building literally fills up with product…which attenuates the signal from the wireless APs little by little. The majority of my WiFi experience is in hospitals, very old buildings and corporate office space where the RF signal only covers about 40-50′ at max and sometimes much smaller (hospitals). The biggest surprise in this facility is the propagation of the RF signal, there’s literally too much RF signal. I’ve had to turn off quite a few AP radios especially in the 2.4Ghz (802.11b/g) band to keep the channel interference to a minimum.

While RF coverage issues are no surprise to anyone working in the industry I have had an interesting time tracking down an idle timeout issue between the Motorola handheld scanners and the IBM AS400 host which runs our warehouse inventory system. The default idle user timeout on the Cisco WLC 5508 is 5 minutes, so I had to changed that parameter to 43200 seconds (12 hours). I also changed the session timeout to 86400 seconds (24 hours). Our employee shifts run 10 hours so these settings looked like they would work – and they do work, except they didn’t work for everyone. In my initial testing I found that I could login from one side of the distribution center, walk to the other side of the distribution center (6 minute walk – it’s that big) and I could pickup my session where I left off without issue even with the handheld going to sleep as I physically moved through the distribution center. I thought the problem was solved and moved onto other issues only to eventually hear that some users were still reporting that they were being “kicked out” – but not everyone.

It was a very specific set of users… those working the receiving dock and the wholesale racking. What was the common denominator? These users would occasionally put down their Motorola handhelds for upwards of 10-15 minutes as they unloaded a truck or prepared to put away a pallet. When they tried to use the handheld it would prompt that their AS400 session had ended and they would need to log in again.

I ran a few tests from the office space in the distribution center and quickly determined that if I was idle longer than 12 minutes I would generally get disconnected from my AS400 session. Was this issue coming from the wireless network or the AS400 host itself? I setup a quick and easy control test. I enabled telnet on one of my Linux jump boxes and established a connection from my test MC9190. After logging into my Linux jumpbox I sat the device on my desk for 25 minutes with no input from me. The Motorola MC9190 went to sleep in about 30 seconds, 25 minutes later I picked up the device, waited for it to power up and connect to the network and found my telnet session still working without issue so this test essentially eliminated the wireless network.

It turns out that the AS400 uses a TCP keepalive to determine if a user is still connected to the system. I’m guessing that since the handheld is sleeping it’s not responding to the keepalive packets so the host system is eventually closing the connection to the handheld. I have some inquires into our AS400 team and IBM to validate my theory.

Cheers!

Update: Sunday October 25, 2015

We were able to update the session keep-alive parameter on the IBM AS400 host. This value defaults to 10 minutes. So every 10 minutes the host will check if the session is still valid by sending a keep-alive packet. If the IBM AS400 host doesn’t get a response to the check it will logoff the session. We increased this value to 20 minutes and that’s helped greatly.