The past two weeks have been an interesting blur thanks to GeoTrust revoking the Aruba certificate securelogin.arubanetworks.com which is used in the captive portal for all Aruba APs, including Aruba ClearPass. The problem started when I received a notification from Aruba on September 9th with the subject line of “Aruba Support Advisory ARUBA-SA-20160908-01 :: ArubaOS Default Certification Revocation”. Unfortunately I didn’t get around to really reading the notification until September 12th. And wouldn’t you know I got my first call about a problem with guest wireless the next day on September 13th.
The Aruba notification cited the following two articles;
- Aruba products contain compromised HTTPS certificate
- Cryptographic Key Reuse Remains Widespread In Embedded Products
In summary, a third party named SEC Consult posted a large number of private keys to GitHub including the private key for securelogin.arubanetworks.com which they had extracted from an Alcatel-Lucent OmniAccess Wireless Access Point. It just happens that Aruba is the OEM for the Alcatel-Lucent OmniAccess product line. With the private key public, it’s fairly easy for anyone to perform a man-in-the-middle attack and eavesdrop on supposedly secure traffic. Shortly afterwards GeoTrust revoked the certificate which will now cause browsers to throw an error when using the certificate for guest registration or captive portal.
This will only happen for browsers which have recently updated their CRL list. If we check the CRL we’ll find the certificate with serial number 01DA52.
[root@centos ~]# wget http://gtssldv-crl.geotrust.com/crls/gtssldv.crl
--2016-09-28 20:15:01-- http://gtssldv-crl.geotrust.com/crls/gtssldv.crl
Resolving gtssldv-crl.geotrust.com... 2600:1400:a:18f::1abd, 2600:1400:a:18b::1abd, 23.50.69.163
Connecting to gtssldv-crl.geotrust.com|2600:1400:a:18f::1abd|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5150 (5.0K) [application/pkix-crl]
Saving to: `gtssldv.crl'
100%[======================================>] 5,150 --.-K/s in 0s
2016-09-28 20:15:01 (189 MB/s) - `gtssldv.crl' saved [5150/5150]
[root@centos ~]# openssl crl -inform DER -text -noout -in gtssldv.crl
Certificate Revocation List (CRL):
Version 2 (0x1)
Signature Algorithm: sha1WithRSAEncryption
Issuer: /C=US/O=GeoTrust Inc./OU=Domain Validated SSL/CN=GeoTrust DV SSL CA
Last Update: Sep 28 23:43:00 2016 GMT
Next Update: Oct 8 23:43:00 2016 GMT
CRL extensions:
X509v3 Authority Key Identifier:
keyid:8C:F4:D9:93:0A:47:BC:00:A0:4A:CE:4B:75:6E:A0:B6:B0:B2:7E:FC
X509v3 CRL Number:
164147
Revoked Certificates:
...
Serial Number: 01DA52
Revocation Date: Sep 7 17:04:44 2016 GMT
...
Signature Algorithm: sha1WithRSAEncryption
58:20:83:3f:5c:cf:31:0f:69:9b:e5:af:dd:ca:2f:b9:e1:42:
b0:bb:68:f3:b8:e8:8c:e2:b1:17:0b:bf:5a:d0:a2:65:3c:1f:
18:bf:93:3a:55:ea:25:c9:da:ba:3b:ed:b6:c7:67:ae:33:b2:
34:40:27:b9:5c:76:8a:3a:8d:4e:8a:9c:d3:61:a7:ed:01:a0:
94:fe:ab:7d:96:79:cb:4e:a5:bb:2e:fb:3c:96:52:3e:ae:e9:
9a:d9:0b:59:3a:c0:f6:45:10:3e:f2:b4:d1:79:63:0b:41:ed:
d4:70:d6:26:67:1d:ab:7a:6e:c8:8d:6a:0e:df:17:ff:b3:e6:
a7:dd:6c:5f:fe:41:79:0c:09:51:cd:6d:7d:33:2a:0b:48:a2:
5c:5c:5c:06:f3:6b:d6:b9:af:14:34:3d:7d:b9:85:36:32:38:
c1:70:50:9d:13:01:ab:b1:de:78:0d:10:24:f2:56:3d:d2:77:
93:d1:70:b8:47:78:a6:22:aa:6a:95:af:49:cb:bc:f6:1e:dd:
1b:0e:bd:10:54:09:39:93:4a:10:a3:1f:76:99:12:47:35:0d:
37:95:6e:fc:a1:b5:e1:f0:d4:f7:96:3e:52:44:e4:69:d9:64:
37:ec:15:50:fd:e8:f3:06:3b:ee:c1:6a:ee:0a:01:fc:fc:e0:
7e:92:d8:0c
We had to replace the certificate, but how do you replace the certificate on almost 600+ Aruba Instant AP Virtual Controllers? That’s where we looked at utilizing Airwave Management Platform (AMP), and the process appeared to be pretty straight forward although like almost everything with technology we quickly ran into problems. We started by getting a new public certificate and combined the certificate, intermediate certificate and root certificate all into a single file. We uploaded that file to AMP and then went through the multiple groups within Airwave to enable the new certificate. And we quickly thought, that was too easy and it was. While a number of groups worked and the VCs in those groups were getting the new certificate, a large number of groups were not working and those VCs were not getting the new certificate. We turned to Auba TAC should guided us through the following steps 1) upgrade of AMP from 8.0.9.2 to 8.0.10, 2) upgrade of the IAPs from 6.3.1.2-4.0.0.4 to 6.4.2.6-4.1.3.1, 3) second upgrade of AMP from 8.0.10 to 8.2.2. We finally discovered that the following was missing from the template; %captive_portal_cert_checksum%. Without this line in the configuration template AMP will not push down the certificate to the IAPs.
Cheers!
References:
ArubaOS Default Certificate Revocation FAQ – Instant
Image Credit: Eliseeva Ekaterina