Michael McNamara

technology, networking, virtualization and IP telephony

Adopting US Access Ports in GB – Ooppss

by

I ran into another interesting problem today…. we use a combination of Aruba and Motorola wireless equipment. When we have an issue with say the captive portal on a Motorola RFS 4000 it’s pretty easy to take an AP 650 and configure it via DHCP to connect via WISP/WISPE/CAPWAP to the remote controller so you can easily observe the problem first hand. In this case the Motorola RFS 4000 happened to be in Barcelona, Spain although it was configured with a country code of GB (Great Britain). I probably spent the better part of two hours trying to get the AP to adopt to the RFS 4000. I originally thought the problem was related to the AP trying to perform a software upgrade over a 200+ms distance between the AP and WLS but even after I disabled the auto-upgrade feature I would still end up with the following SYSLOG messages;

Nov 20 18:21:59 2014: LED state message WIOS_LED_NO_COUNTRY_0_24G from module DOT11 : %DIAG-6-NEW_LED_STATE:
Nov 20 18:21:59 2014: LED state message RADIO_ALL_LED_OFF from module DOT11 : %DIAG-6-NEW_LED_STATE:
Nov 20 18:21:59 2014: Radio 'ap650-981XXX:R1' changing state from 'Initializing' to 'Off(no country-code)' : %RADIO-5-RADIO_STATE_CHANGE: ff(no country-code)'
Nov 20 18:21:49 2014: RFS-4000 : %AP-6-ADOPTED: Access Point('ap650-981XXX'/'AP650'/5C-0E-8B-98-XX-XX) at rf-domain:'default' adopted and configured. Radios: Count=1, Bss: 5C-0E-8B-31-XX-XX|

I finally realized that the AP650s I had were US models and not WW models. I was able to take an AP300 (WSAP-5110-100-WW) and configure it to connect to the RFS 4000 via DHCP option 189 and ultimately solve the puzzle around the captive portal issue (future blog post).

Cheers!

Avaya SPB Implementations – Packet Pushers

by

I get a lot of correspondence asking if Avaya’s SPB is real and what if any adoption is the industry seeing around SPB. While there are a few different vendors providing SPB based solutions it’s pretty clear that Avaya is leading the pack.

Dominik, one of our discussion forum moderators, was a recent guest to the Packet Pushers podcast in Show 210 – SPB Implementation Fundamentals where he discussed his SPB implementations with Ethan and Greg and Ricki Cook.

I’d recommend you give a listen if you are at all interested in how SPB works.

Cheers!

Internet Service Providers – Source Route Filtering

by

I ran into an interesting problem this past week after I made a fairly small change on one of my border BGP routers. We upgraded one link from a 100Mbps circuit to a 1000Mbps circuit and it was decided that we should use this link as our preferred path for all traffic egressing our network. We had previously been using a Comcast link as the preferred egress path for all traffic but we were going to change that using the BGP local-pref attribute. While those changes themselves were relatively straight forward and went off without a hitch there was an unintended consequence that stumped me for a few days. Upon making the change we received notification from our external monitoring servers that our Level3, Comcast and Verizon WAN IP interfaces had gone unreachable, previously there were reachable from our two external monitoring servers (Linux servers hosted in a VSP on opposite coasts of the United States). The alarm was a surprised but when I checked the Cisco ASR1001 interfaces everything was up and running although sure enough the two Linux servers were unable to ping the WAN IP interfaces on the border router for the Level3, Comcast and Verizon circuits. The two Linux servers were able to ping the WAN IP interface of the AT&T circuit. If I issued a ping from the Cisco ASR 1001 itself it had no issues pinging the Linux servers. If I tried to ping the two Linux servers from the router by sourcing the traffic from the previously mentioned WAN IP inetrfaces that would fail as well. That was odd I thought, what was going on there? Prior to this change the BGP local-pref preferred the Comcast circuit for all outbound traffic as visualized below.

Multihomed BGP Router 1Once we made the BGP local-pref change all IP traffic was egressing the AT&T circuit as visualized below.

Multihomed BGP Router 2

There was never a problem reaching any of the ARIN IP address blocks that we were advertising via BGP the problem was isolated to just the WAN IP interfaces of the other Internet Service Providers.

The problem turned out to be that traffic to the WAN IP address was ingressing the circuit that the IP address was assigned to but it would egress the AT&T circuit due to the BGP local-pref statement. I’m guessing that AT&T is filtering the traffic on ingress checking for traffic sourced from an IP address block that has no business coming from that link and was dropping the traffic.

Multihomed BGP Router 3So an ICMP packet to the Comcast WAN IP address would ingress the Comcast interface and would egress the AT&T interface with a source IP address of the Comcast WAN interface. That packet would hit the AT&T head-end router which would discard any packets not sourced from the a valid ARIN IP address block belonging to that link, similar to Reverse Path Forwarding. I was able to verify this by placing a pair of static routes on the router using the Comcast circuit as a return path and with that the two Linux servers were now able to ping all the WAN IP interfaces. I’m guessing that while AT&T does some source route filtering, Comcast isn’t doing any.

It think it’s great that AT&T is filtering their inbound traffic for valid source IP blocks, it definitely helps prevent IP spoofing.

The confusion came when I did a debug ip icmp and later a debug ip packet 100 detail and observed no ICMP traffic coming from either of the two Linux servers on the Cisco ASR1001. I had a ticket open with Cisco TAC and they were also unable to explain the oddity. I’m curious if this was something to-do with CEF and might I need to enable no ip route-cache on the specific interfaces?

Cheers!

Planetside 2 – Gaming With Scale

by

I’ve been playing multi-player first person shooters since the original Doom more than 20 years ago. Last year I was introduced to Planetside 2, a game that truly brings scale to the multi-player first person shooter. I’ve played quite a few multi-players games including Battlefield 2, Counter-Strike, Day Of Defeat, Fontlines, Battlefield 2142, Battlefield 4, Call of Duty, Left for Dead, Rising Storm, Medal of Honor, Red Orchestra 2, Team Fortress 2, etc. And while most of those were great games and provided me plenty of entertainment you usually couldn’t play with any more than 63 other players on the same server. In Planetside 2 you can play on a continent (map) with upwards of 2,000 other players. There’s something to be said about seeing 100+ enemy soldiers running down the hillside at you.

Planetside 2 is free to play if your at all interested, although you’ll need a fairly powerful desktop/laptop to truly enjoy the game. I should also mention that you’ll probably need more than a few hours to download the entire game as it’s reigns in at almost 15GBs in size.

Cheers!

Storage Area Networks

by

I’ve recently found myself dealing with a lot of storage area network issues. It’s my team that has the privilege of maintaining 22 different Fiber Channel Storage Area Network (SAN) switches with multiple independent and redundant SAN fabrics. I’ve recently run into a number of situations where improper cabling and FC SAN switch zoning were the root cause to some very visible application outages. A SAN provides “block” level access to a multitude of devices including disk arrays such as the EMC VNX 5300 and 7500, and tape drives such as the HP MSL 4048 tape library. Storage Area Networks provide centralized storage and usually play an important role in cluster solutions due to their ability to share storage between multiple servers or hosts. If you have a traditional (non-VSAN) VMware ESXi cluster you have either a Fiber Channel or iSCSI SAN providing the VMware datastores between the ESXi hosts.

A few weeks ago I had the opportunity to expand the SAN fabric by installing a pair of Brocade 5100 FC SAN switches. You need to follow a pretty simple process to ISL a FC SW to an existing SAN fabric.

  1. You must make sure that the new switch has a unique Domain ID in relation to all the other switches in the fabric
  2. You must make sure make that the new switch doesn’t have an active ZONE configuration
  3. With those two steps completed you only need to physically connect the switches.

The switches will automatically configure the FC ports as E-Ports and establish an Inter Switch-Link  (ISL) trunk between the two FC switches. The ZONE configuration will be copied over to the new switch so there’s very little to-do beyond configuring an IP address for management.

Whether you have a large storage network or you just have a single FC switch you might want to look at Brocade SAN Health.

BrocadeSANHealth

I was very surprised by the level of detail that the tool was able to capture, and the Visio diagram was awesome!

I also took the opportunity to do some housekeeping by going back through all the FC switches and make some general configuration changes enabling SNTP, setting the timezone and configuring SYSLOG logging. I can’t tell you how maddening it is trying to troubleshoot a problem where the log time stamps are all wrong. Here are the CLI commands that I used to make those configuration changes;

tsTimeZone US/Eastern
tsclockserver 1.2.3.4
syslogdipadd 1.2.3.4

While changing the timezone the switch reported that I needed to reboot the switch for the change to take affect. I have yet to reboot the switch but the time is reporting the correct offset from GMT. This message was consistent across all versions of Brocade’s Fabric OS including v6.x and v7.x.

Cheers!

Recent Posts

Categories

SUBSCRIBE BY EMAIL