Michael McNamara

technology, networking, virtualization and IP telephony

Conclusion – 30 Days of Peak

by

It was back on November 24, 2014 that I accepted Greg’s challenge to write 30 blog posts in 30 days. I’ve made it to day 30 and I’m still here, although I’m honestly more than a little tired and burnt out. Working for a retail company makes the Christmas holidays extremely stressful – the holiday peak is extremely important for every retailers balance sheet so there’s extra scrutiny around any technical issues whether they are actual or just perceived. I’ve heard horror stories of past holiday peaks so it appears that this holiday has been pretty successful. We had no major issues in our stores or on our websites and thankfully we didn’t have any major issues on Black Friday or Cyber Monday. We had a few challenges in one of our distribution centers thanks to ICMPv6 traffic flooding the network. Overall I’d like to think that I had something to-do with that success, probably not at as much as I’d like to take credit for. I’ve got a great team working for me so I need to credit them as well. I had been at my previous employer for 17 years and had the network infrastructure pretty well greased and running without issues, I’ve come to the realization that it’s going to take me a lot longer than 12 months to get the network infrastructure where I’d like it to be with my new employer.

As I mentioned when I started this challenge that I thought there might only be a handful of decent posts.

There were a few posts that I wrote but after I thought about them for a while I decided not to post them. I now work for a publicly traded company so I need to be mindful of not inadvertently publishing something that isn’t public knowledge – such as sales figures. I also work for a global retailer that’s been the target of some very intricate hacking attempts so I need to be mindful of what I reveal about our security posture. I have some stories from consulting engagements that would make great posts, but again I need to be mindful of my clients privacy even if I don’t name them outright. Especially if they are repeat clients which most of mine are.

Here are my favorite posts from the past 30 days;

Cheers!

Image Credit: Ted C.

Note: This is a series of posts made under the Network Engineer in Retail 30 Days of Peak, this is post number 30 of 30. All the posts can be viewed from the 30in30 tag.

Website Monitoring with Bash and Nagios Plugins

by

It’s no surprise that I need to know when our websites are down, but I also need to know why they went down. Often the redundancy will kick in and the website will quickly recovery. However, the question remains why was the website down? Was there a circuit failure, a router failure, a load-balancer failure, a web server failure, an application server failure, a database failure? While you can glean a lot of information from the log data generated by the routers, firewalls, switches, load-balancers and web servers sometimes there are gaps in that data. A few months ago I put together a quick bash script that calls a few Nagios plugins to help me gather some data points in the event that I needed to look back in time, after the fact, to determine what had caused an outage or failure. I decided to stand up a few Linode Linux servers spread out across a number of Data Centers around the world. While there are dozens if not hundreds of commercial solutions for website monitoring but I wanted something cheap in which I had complete control over and writing this script took all of 2 hours one afternoon.

The script will run every 60 seconds and will call the origin web server via an HTTP call and validate that it’s returning the proper HTML content. If the server fails to answer the first HTML call or response doesn’t contain the prerequisite content the script will wait and try a second time. If the second HTTP call fails the script will then log that fact and it will try a PING to verify that it can reach the web server. If the PING fails, the script will kick off a traceroute using mtr to try and isolate the location of the problem.  A second script performs ICMP pings every 60 seconds to every piece of our public network infrastructure including the firewalls and load-balancers across our multiple Data Centers from multiple public Internet points.

The combination of the data points from both scripts, being run in multiple Data Centers around the world made it relatively easy to quickly determine what had transpired during an event. In one case we were alerted to a peering issue between NAC and Level3. In another event we observed a complete disconnect between NetworkLayer/SoftLayer and Comcast between 1AM and 2AM one night – I’m guessing that was some type of scheduled maintenance, and they didn’t have BGP configured properly. There were a few times though when the script would alert that everything was down but only from a single Data Center, this often indicated that there was a problem with the Internet peers that connected that Data Center to the Internet in general. It wasn’t a fool proof solution by any means but it gave me the data points I needed and the freedom to adapt as needed.

You can download the entire script from the link.

#!/bin/bash
#
# Filename: /usr/local/monitor/monitor.sh
#
# Purpose:  Monitor the availability of several websites and report their
#           availabilty. This script leverages several Nagios plugins to
#           help simplify the collection of data.
#
# Language: Bash Script
#
# Author:   Michael McNamara
#
# Verzion:  0.9
#
# Date:     Oct 26, 2014
#
# License:
#           Copyright (C) 2014 Michael McNamara (mfm@michaelfmcnamara.com)
#
#           This program is free software: you can redistribute it and/or modify
#           it under the terms of the GNU General Public License as published by
#           the Free Software Foundation, either version 3 of the License, or
#           (at your option) any later version.
#
#           This program is distributed in the hope that it will be useful,
#           but WITHOUT ANY WARRANTY; without even the implied warranty of
#           MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#           GNU General Public License for more details.
#
#           You should have received a copy of the GNU General Public License
#           along with this program.  If not, see <http://www.gnu.org/licenses/>
#
# Changes:
#           Nov 11, 2014 add lock file checking to prevent multiple instances
#           Oct 31, 2014 added code to retry the HTTP_CHECK before alarm
#           Oct 30, 2014 added additional websites to query
#           Oct 27, 2014 cleaned up script/updated documentation
#
# Requirements:
#
#           Nagios check_icmp plugin
#           Nagios check_http plugin
#           Nagios check_dns plugin
#           http://nagiosplugins.org/
#
# Notes:
#        Command Line Reference;
#          ./monitor.sh
#
#

# Declare Variables
SENDMAIL="/bin/mail"

CHECK_HTTP="/usr/local/monitor/check_http"
CHECK_FPING="/usr/local/monitor/check_fping"
CHECK_DNS="/usr/local/monitor/check_dns"

MTR="/usr/sbin/mtr"
LOG="/usr/local/monitor/monitor.log"
LOCKFILE="/tmp/monitor.tmp"
LOCATION="New York, NY"

MAIL_TO="root"
MAIL_SUBJECT="HTTP: Web Application Status Report ($LOCATION)"

#
### SITE SPECIFIC INFORMATION <<<<< YOU SHOULD EDIT THE LINES BELOW
#
# IPS = List of webservers by FQDN or IP address
IPS=( webserver1.acme.com webserver2.acme.com webserver3.acme.com)

# HOSTS = The FQDN of the web property that resides on the webserver
HOSTS=( www.brandone.com www.brandtwo.com www.brandthree.com )

# URLS = The path to be appended to the FQDN of the web property
URLS=( /brand1/index.jsp /brand2/index.jsp /brand3/index.jsp )

# CONTENTS = A regex containing some text that should be found on
#            the webpage for each brand or web property.
CONTENTS=( "Brand One" "Brand Two" "Brand Three" )
#
# <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 

#################################################################### 
# M A I N    P R O G R A M 
#################################################################### 
# LETS WAIT FOR A LITTLE SO WE'RE NOT FIRING AT THE TOP OF THE MINUTE 
sleep 15 

# LETS CHECK TO SEE IF THERES ALREADY A COPY RUNNING 
if [ -e ${LOCKFILE} ] && kill -0 `cat ${LOCKFILE}`; then     
    echo "already running"     
    exit 
fi 

# SETUP A TRAP INCASE WE EXIT PREMATURELY 
trap "rm -f ${LOCKFILE}; exit" INT TERM EXIT 

# LETS CREATE A LOCKFILE 
echo $ > ${LOCKFILE}

# LETS ITERATE OVER THE WEBSERVERS ($IPS[])
for (( i = 0; i < ${#IPS[@]}; i++ )) do     
    # LETS TRY A QUICK HTTP CALL AND SEE WHAT WE GET
    RESULT1="`${CHECK_HTTP} -I ${IPS[$i]} -H ${HOSTS[$i]} -u ${URLS[$i]} -s ${CONTENTS[$i]}`"     

    # LETS CHECK THE RESULT
    if [[ $RESULT1 =~ "OK" ]]     then
         # IF THE RESULT WAS OK THEN LOG THE RESULT
         echo "$(date) OK ${IPS[$i]} ${HOSTS[$i]} $RESULT1" >> $LOG
    else
        # IF THE RESULT WAS BAD LETS DO MORE, FIRST WAIT A LITTLE
        sleep 10

        # LOG THAT WE FAILED THE FIRST CHECK
        echo "$(date) FAIL ${IPS[$i]} ${HOSTS[$i]} $RESULT1" >> $LOG

        # ATTEMPT A SECOND HTTP CALL AND SEE WHAT WE GET
        RESULT2="`${CHECK_HTTP} -I ${IPS[$i]} -H ${HOSTS[$i]} -u ${URLS[$i]} -s ${CONTENTS[$i]}`"

        # LETS CHECK THE RESULT
        if [[ $RESULT2 =~ "OK" ]]
        then
            # IF THE RESULT WAS OK THEN LOG THE RESULT
            echo "$(date) RETRY OK ${IPS[$i]} ${HOSTS[$i]} $RESULT2" >> $LOG
        else
            # IF THE RESULT WAS BAD LETS DO MORE, FIRST LOG AND EMAIL
            echo "FAIL RETRY ${IPS[$i]} ${HOSTS[$i]} $RESULT2" | $SENDMAIL -s "$MAIL_SUBJECT" $MAIL_TO
            echo $(date) RETRY FAIL ${IPS[$i]} ${HOSTS[$i]} $RESULT2 >> $LOG

            # LETS TEST AN ICMP CALL TO THE WEBSERVER TO VALIDATE ITS NOT A NETWORK ISSUE
            PING1="`${CHECK_FPING} ${HOSTS[$i]} -n 3`"

            # LETS CHECK THE RESULT
            if [[ $PING1 =~ "OK" ]]
            then
                # IF THE RESULT WAS OK THEN LOG THE RESULT
                echo $(date) OK ${IPS[$i]} ${HOSTS[$i]} $PING1 >> $LOG
            else
                # IF THE RESULT WAS BAD THEN LOG AND EMAIL AND COLLECT A TRACEROUTE
                echo "FAIL RETRY PING ${IPS[$i]} ${HOSTS[$i}} $PING1" | $SENDMAIL -s "$MAIL_SUBJECT" $MAIL_TO
                echo $(date) FAIL ${IPS[$i]} ${HOSTS[$i]} $PING1 >> $LOG
                `${MTR} --no-dns -rc 5 ${IPS[$i]} >> $LOG`
            fi
        fi
    fi
done

# LETS WAIT FOR A FEW SECONDS
sleep 10

# LETS CLEAN UP AND REMOVE THE LOCKFILE
rm -f ${LOCKFILE}

# THATS ALL FOLKS!
exit 0

Cheers!

Note: This is a series of posts made under the Network Engineer in Retail 30 Days of Peak, this is post number 29 of 30. All the posts can be viewed from the 30in30 tag.

Image Credit: michele de notaristefani

Vizio 39″ LED TV – Google Chromecast

by

It just happened that my 30″ Sony WEGA TV finally gave up the ghost this past week. That beast weighed at least 150 lbs, and I wasn’t looking forward to getting it out of the wall hutch I had built almost 10 years ago. I had to ask both my wife and 14 year for help getting it out of the hutch and out into the garage.

I had to make a quick purchase and I was limited by the dimensions that the wall hutch would allow, a width of 38 3/4″. I did some research and landed on the Vizio 39″ LED TV sold by Best Buy – yes, I still frequent Best Buy and enjoy having a brick and mortar store that I can run down to in a pinch and pickup an item. I decided to pickup a third Google Chromecast already having two others elsewhere in the house.

HDMI-CEC

Google-ChromecastWith Consumer Electronics Control (CEC) you can simultaneously turn on your TV and Chromecast and even change the TV to the correct HDMI input without ever touching your TV remote. I had to enable this feature in the Vizio TV, it was automatically enabled in my 40′ Samsung LED TV, but it makes using the Chromecast very user friendly. You just select the Chromecast on your smart phone, tablet or laptop and it turns on the TV, changes to the correct HDMI input and starts showing your content.

The Chromecast supports 2.4Ghz 802.11bg wireless networks although a 5Ghz version is rumored to be in the works. There’s a deployment guide from Cisco that details how the Chromecast works and what settings are needed in an enterprise wireless network. There’s a little known tidbit about which data rate Multicast traffic is transmitted at;

Multicast applications, such as Chromecast, require special consideration when being deployed over a wireless network because a multicast in 802.11 is sent out as a broadcast so that all clients can hear it. The actual data rate used by the AP in order to transmit the Chromecast frames is the highest mandatory rate configured within that band. For 2.4 GHz, the default rate is 11 Mbps.

In order to optimize the delivery of these frames, it is important to tune the 802.11 data rates within the controller to allow multicast to be delivered at the highest rate that the coverage model of the network can support. For networks with a low density of APs, it may be necessary to keep the data rates at the default. For a network that does not have any requirement to support 802.11b clients, tuning the data rate to 12 Mbps mandatory and lower rates disabled will help to reduce multicast airtime utilization.

I’ve run into similar issues in my enterprise wireless network with Apple TVs as opposed to Google Chomecast, but the same issues apply.

There’s a great article from How-To Geek that details how you can stream your browser tabs and even your entire desktop to the Chromecast. There’s also a dizzying array of applications that now support Chromecast.

Cheers!

Note: This is a series of posts made under the Network Engineer in Retail 30 Days of Peak, this is post number 28 of 30. All the posts can be viewed from the 30in30 tag.

How ICMPv6 Multicast Listener Reports almost spoiled Christmas

by

If you’ve been following me recently you might recall that I’ve been chasing an issue with a Motorola WS5100 running v3.3.5.0-002R experiencing high CPU utilization. The problem came to a head this weekend and here’s my quick account of the experience.

The WS5100 would intermittently come under extreme load for 5-30 minutes, so much load that ultimately the entire wireless network would collapse as the Access Ports started experiencing watchdog resets and would just continually reboot themselves. This problem would come and go throughout the day or night, we could go 12 hours without an issue and then go the next 12 hours with issues every 30 minutes. The problem was affecting both the primary and secondary WS5100 so I eliminated the hardware almost out of the gate. I have first hand experience running v3.3.5.0-002R software on a large number of WS5100s and have never had an issue with that software release so I really didn’t suspect the software. This wireless solution had been in place for more than 18 months without any major issues or problems. The local engineers reported that there had been no changes, no new devices. So what was causing this problem? I immediately suspected an external catalyst but how would I find it?

As with most highly technical problems it wasn’t until I could get my hands on some packet traces and I had time to dissect those packet traces that I could start to fully understand and comprehend what was actually going on.

Topology

A pair of Motorola WS5100 Wireless LAN Switches with 30 AP300 running software release v3.3.5.0-002R in a cluster configuration with one running as primary and the other running as secondary. The network was comprised of a single Cisco Catalyst 4500 with around ten individual Cisco Catalyst 2960S switches at the edge each trunked to the core in a simple hub and spoke design. The entire network was one single flat VLAN. The WS5100s were attached to the Cisco Catalyst 4500 via a single 1Gbs interface, one arm router style. The peak number of wireless devices was around 200, the total number of MAC addresses on the network was around 525 (this includes the wireless devices).

Symptoms

The initial problem report centered around poor wireless performance and sure enough I quickly found 30-40% packet loss while just trying to ping the WS5100. When I finally got logged into the WS5100 I could see that the CPU was running at 100%. The SYSLOG data showed me that the APs were rebooting because of watchdog timeouts. PTRG was showing me that here was a huge traffic surge being received from the WS5100. I quickly realized that the traffic spikes in the graph correspond to events that users were experiencing problems.

TrafficStorm

Packet Traces

I directed the team to setup a SPAN port to capture the traffic that was flowing between the WS5100 and the Cisco Catalyst 4500 switch. This would provide me a better idea of what was actually on the wire and might provide a clue as to what was transpiring. The team setup Wireshark to continually capture to disk using a 100MB file size and allowing the file to wrap 10 times for a total of 1GB of captured data. The next time the problem occurred I was alerted within 15 minutes by the help desk and users but I found that we missed the start of the event. There was so much traffic Wireshark only had the past 3 minutes available on disk so we had to increase the filesize to 300MB and the number of wrap files to 25 giving us a total capacity of 7.5GB. That configuration would eventually allow me to capture the initial events along with the time needed to get to the laptop and copy the data before it was overwritten. While I waited for the problem to occur I took to setting up SWATCH to alert myself and the team when the problem started so we could quickly gather all the data points during the start of the event.

WireShark-ICMPv6MulticastListenReport

Using the data from the packet traces we were able to identify and locate two HP desktops that were apparently intermittently flooding the network with ICMPv6 Multicast Listener Reports.

We removed those HP desktops from the network and everything has been stable since.

Analysis

Here’s the current working theory which I believe is fairly accurate. The HP desktops were intermittently flooding the network with ICMPv6 Multicast Listener Reports. Those packets were reaching the WS5100 and because the network at this location is a single flat VLAN the WS5100 needs to bridge those packets over to the wireless network. It does this by encapsulating them in MiNT in a fashion very similar to CAPWAP  or LWAPP. The issue here is the number of packets and the number of access points or access ports. In this case we had 30 APs connected to the WS5100 so let’s do some rough math;

41,000 ICMPv6 Multicast packets * 2 HP desktops = 82,000 packets * 30 APs = 2,460,000 packets

This explains the huge amount of traffic the WS5100 is transmitting. For every ICMPv6 Multicast packet (or broadcast packet for that matter) received by the WS5100, it needs to encapsulate and send a copy of that packet to each and every AP. If there are 30 APs then the WS5100 needs to copy each and every packet 30 times. Now multiply that by the number of ICMPv6 packets that were being received by the WS5100 and you have a recipe for disaster.

A quick search of Google will reveal a number of well documented issues with Intel NICs.

The HP desktops turned out to be HP ProDesk 600 G1s running Windows 7 SP1 with Intel I217-LM NICs driver v12.10.30.5890 with sleep and WoL enabled.

Summary

There were a few lessons learned here;

  1. The days of the single flat network are gone. It’s very important to follow best practice when designing and deploying both wired and wireless infrastructures. In this case if the wireless infrastructure had dedicated VLANs both for the wireless client traffic and for the AP traffic this problem would have never impacted the WS5100. It may have impacted the Cisco Catalyst 4500 somewhat but it wouldn’t have caused the complete collapse of the wireless infrastructure. Unfortunately in this case everything was on VLAN 1, wired clients, APs, wireless clients, servers, IP phone systems, routers, everything.
  2. The filtering of IPv6 along with Multicast and broadcast traffic from the wireless infrastructure is especially important. I posted back in September 2013 how to filter IPv6, multicast and broadcast packets from a Motorola RFS7000, the same applies to the WS5100. Unless you are leveraging IPv6  in your infrastructure, or have some special multicast applications you should definitely look into filtering this traffic from your wireless network.
  3. Validate those desktop and laptop images, especially the NIC drivers and WNIC drivers. In the early days of 802.1x I can remember documenting a long list of driver versions and Microsoft hotfixes required for Microsoft Windows XP (pre SP2) in order to get 802.1x authentication (Zero Wireless Configuration) to work properly.

Conclusion

Wireshark saved this network engineer’s holiday – Thanks!

Cheers!

Note: This is a series of posts made under the Network Engineer in Retail 30 Days of Peak, this is post number 27 of 30. All the posts can be viewed from the 30in30 tag.

Swatch – Simple Log Watcher

by

It’s a wonder the odd and bizarre problems that seem to find me. Straight from the front lines I had an issue with a Motorola WS5100 v3.3.5.0-002R falling down at the most inopportune time of the retail calendar. While the original problem appeared on December 17 it returned last night to spoil the weekend.

In the process of trying to understand the problem and come up with a solution I wanted to have better visibility and alerting when the problem actually occurred, I didn’t want to incur the delay that would involve the users calling the help desk and the help desk calling me. Thankfully there is a SYSLOG message recorded when an Access Port experiences a watchdog reset so I had a log message now I needed to find a way to alert on that message.

That’s where I turned to swatch, a handy little utility that will monitor log files for regular expressions and then take whatever action, such as ringing the console or sending an email message is configured. I installed swatch with relative ease thanks to yum and then set out to configure it appropriately.

I created the following configuration file;

#/etc/swatchrc

# swatchrc - define regular expressions and generate alerts when matches are found in logs
#            daemon is started from /etc/cron.d/swatch

# Motorola AP300 - malfunctioning AP ignore events from this device

ignore /00-A0-F8-ZZ-ZZ-ZZ/

# Motorola WS5100 Access Port Adoption Errors Reboot/Watchdog events

#Dec 20 07:53:07 ACME-WLS1 %CC-6-APREADOPTREASON: AP 00-A0-F8-XX-XX-XX readoption reason: ColdBoot/Watchdog
#Dec 20 07:53:25 ACME-WLS1 %CC-6-APREADOPTREASON: AP 00-A0-F8-XX-XX-XX readoption reason: Link failed

# Let's look for the phrase readoption and we'll alert of that text

watchfor /readoption/
        exec "echo '$_' | mail swatch -s 'SWATCH: Motorola WS5100 Adoption Issue' "
        threshold track_by=$6,type=limit,count=1,seconds=60
        echo=red
        bell 5

#end

In the swatch configuration I used the mail aliase of swatch so I edited the /etc/newaliases file to make sure that the entire team would receive the alert;

#
#  Aliases in this file will NOT be expanded in the header from
#  Mail, but WILL be visible over networks or from /bin/mail.
#
#       >>>>>>>>>>      The program "newaliases" must be run after
#       >> NOTE >>      this file is updated for any changes to
#       >>>>>>>>>>      show through to sendmail.
#

# Basic system aliases -- these MUST be present.
mailer-daemon:  postmaster
postmaster:     root

# General redirections for pseudo accounts.
bin:            root
daemon:         root
adm:            root
...
...
...
swatch:         root,mike,john,dan,tom

If the problem is extremely important I’ll usually add the the email SMS text message gateway for my provider. This way I’ll get both an email message and an SMS text message alerting me to the problem.

# Verizon SMS Text Messaging 123456789@vtext.com
# AT&T SMS Text Messaging 123456789@txt.att.net
# T-Mobile SMS Text Messsaging 123456789@tmomail.net
# Sprint SMS Text Messaging 123456789@messaging.sprintpcs.com

I made sure to recompile the aliases file with the newaliases command and then I set off to run swatch in the foreground of my SSH session.

[root@centos /]# swatch -c /etc/swatchrc -p 'tail -f -n 0 /var/log/loc1fac17.log'

*** swatch version 3.2.3 (pid:30643) started at Sat Dec 20 15:54:14 EST 2014

And I waited for the event.

Now I could go about doing some research and due diligence without worrying that I might inadvertently fail to spot the problem.

I’ll let you know how it turned out!

Cheers!

Note: This is a series of posts made under the Network Engineer in Retail 30 Days of Peak, this is post number 26 of 30. All the posts can be viewed from the 30in30 tag.

Recent Posts

Categories

SUBSCRIBE BY EMAIL