It started as it always does with reports of DNS issues, then CDN issues (Cloudflare) and eventually CenturyLink was identified as the culprit, or to be more precise any packets traversing the CenturyLink (Level3) network.

Thankfully Reddit was a great community resource and reports quickly started rolling in on these two threads;

For reasons that still aren’t 100% clear the moderators for r/networking decided to delete the first thread. So the refugees from r/networking went to r/sysadmin to escape the persecution only to have the moderators of r/networking admit their mistake sometime later and un-delete the post.

I’ll admit I was floored when I found the original thread was deleted. There were hundreds of us struggling to source what was actually going on and trying to understand how we could mitigate the impact to our employers and some moderator deletes the thread?!? @$%#

The refugees eventually made their feelings known in a thread titled, META: I guess major news-worthy outages are off topic here?

Cheers!

The script will run every 60 seconds and will call the origin web server via an HTTP call and validate that it’s returning the proper HTML content. If the server fails to answer the first HTML call or response doesn’t contain the prerequisite content the script will wait and try a second time. If the second HTTP call fails the script will then log that fact and it will try a PING to verify that it can reach the web server. If the PING fails, the script will kick off a traceroute using mtr to try and isolate the location of the problem.  A second script performs ICMP pings every 60 seconds to every piece of our public network infrastructure including the firewalls and load-balancers across our multiple Data Centers from multiple public Internet points.

The combination of the data points from both scripts, being run in multiple Data Centers around the world made it relatively easy to quickly determine what had transpired during an event. In one case we were alerted to a peering issue between NAC and Level3. In another event we observed a complete disconnect between NetworkLayer/SoftLayer and Comcast between 1AM and 2AM one night – I’m guessing that was some type of scheduled maintenance, and they didn’t have BGP configured properly. There were a few times though when the script would alert that everything was down but only from a single Data Center, this often indicated that there was a problem with the Internet peers that connected that Data Center to the Internet in general. It wasn’t a fool proof solution by any means but it gave me the data points I needed and the freedom to adapt as needed.

You can download the entire script from the link.

#!/bin/bash
#
# Filename: /usr/local/monitor/monitor.sh
#
# Purpose:  Monitor the availability of several websites and report their
#           availabilty. This script leverages several Nagios plugins to
#           help simplify the collection of data.
#
# Language: Bash Script
#
# Author:   Michael McNamara
#
# Verzion:  0.9
#
# Date:     Oct 26, 2014
#
# License:
#           Copyright (C) 2014 Michael McNamara (mfm@michaelfmcnamara.com)
#
#           This program is free software: you can redistribute it and/or modify
#           it under the terms of the GNU General Public License as published by
#           the Free Software Foundation, either version 3 of the License, or
#           (at your option) any later version.
#
#           This program is distributed in the hope that it will be useful,
#           but WITHOUT ANY WARRANTY; without even the implied warranty of
#           MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#           GNU General Public License for more details.
#
#           You should have received a copy of the GNU General Public License
#           along with this program.  If not, see <http://www.gnu.org/licenses/>
#
# Changes:
#           Nov 11, 2014 add lock file checking to prevent multiple instances
#           Oct 31, 2014 added code to retry the HTTP_CHECK before alarm
#           Oct 30, 2014 added additional websites to query
#           Oct 27, 2014 cleaned up script/updated documentation
#
# Requirements:
#
#           Nagios check_icmp plugin
#           Nagios check_http plugin
#           Nagios check_dns plugin
#           http://nagiosplugins.org/
#
# Notes:
#        Command Line Reference;
#          ./monitor.sh
#
#

# Declare Variables
SENDMAIL="/bin/mail"

CHECK_HTTP="/usr/local/monitor/check_http"
CHECK_FPING="/usr/local/monitor/check_fping"
CHECK_DNS="/usr/local/monitor/check_dns"

MTR="/usr/sbin/mtr"
LOG="/usr/local/monitor/monitor.log"
LOCKFILE="/tmp/monitor.tmp"
LOCATION="New York, NY"

MAIL_TO="root"
MAIL_SUBJECT="HTTP: Web Application Status Report ($LOCATION)"

#
### SITE SPECIFIC INFORMATION <<<<< YOU SHOULD EDIT THE LINES BELOW
#
# IPS = List of webservers by FQDN or IP address
IPS=( webserver1.acme.com webserver2.acme.com webserver3.acme.com)

# HOSTS = The FQDN of the web property that resides on the webserver
HOSTS=( www.brandone.com www.brandtwo.com www.brandthree.com )

# URLS = The path to be appended to the FQDN of the web property
URLS=( /brand1/index.jsp /brand2/index.jsp /brand3/index.jsp )

# CONTENTS = A regex containing some text that should be found on
#            the webpage for each brand or web property.
CONTENTS=( "Brand One" "Brand Two" "Brand Three" )
#
# <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 

#################################################################### 
# M A I N    P R O G R A M 
#################################################################### 
# LETS WAIT FOR A LITTLE SO WE'RE NOT FIRING AT THE TOP OF THE MINUTE 
sleep 15 

# LETS CHECK TO SEE IF THERES ALREADY A COPY RUNNING 
if [ -e ${LOCKFILE} ] && kill -0 `cat ${LOCKFILE}`; then     
    echo "already running"     
    exit 
fi 

# SETUP A TRAP INCASE WE EXIT PREMATURELY 
trap "rm -f ${LOCKFILE}; exit" INT TERM EXIT 

# LETS CREATE A LOCKFILE 
echo $ > ${LOCKFILE}

# LETS ITERATE OVER THE WEBSERVERS ($IPS[])
for (( i = 0; i < ${#IPS[@]}; i++ )) do     
    # LETS TRY A QUICK HTTP CALL AND SEE WHAT WE GET
    RESULT1="`${CHECK_HTTP} -I ${IPS[$i]} -H ${HOSTS[$i]} -u ${URLS[$i]} -s ${CONTENTS[$i]}`"     

    # LETS CHECK THE RESULT
    if [[ $RESULT1 =~ "OK" ]]     then
         # IF THE RESULT WAS OK THEN LOG THE RESULT
         echo "$(date) OK ${IPS[$i]} ${HOSTS[$i]} $RESULT1" >> $LOG
    else
        # IF THE RESULT WAS BAD LETS DO MORE, FIRST WAIT A LITTLE
        sleep 10

        # LOG THAT WE FAILED THE FIRST CHECK
        echo "$(date) FAIL ${IPS[$i]} ${HOSTS[$i]} $RESULT1" >> $LOG

        # ATTEMPT A SECOND HTTP CALL AND SEE WHAT WE GET
        RESULT2="`${CHECK_HTTP} -I ${IPS[$i]} -H ${HOSTS[$i]} -u ${URLS[$i]} -s ${CONTENTS[$i]}`"

        # LETS CHECK THE RESULT
        if [[ $RESULT2 =~ "OK" ]]
        then
            # IF THE RESULT WAS OK THEN LOG THE RESULT
            echo "$(date) RETRY OK ${IPS[$i]} ${HOSTS[$i]} $RESULT2" >> $LOG
        else
            # IF THE RESULT WAS BAD LETS DO MORE, FIRST LOG AND EMAIL
            echo "FAIL RETRY ${IPS[$i]} ${HOSTS[$i]} $RESULT2" | $SENDMAIL -s "$MAIL_SUBJECT" $MAIL_TO
            echo $(date) RETRY FAIL ${IPS[$i]} ${HOSTS[$i]} $RESULT2 >> $LOG

            # LETS TEST AN ICMP CALL TO THE WEBSERVER TO VALIDATE ITS NOT A NETWORK ISSUE
            PING1="`${CHECK_FPING} ${HOSTS[$i]} -n 3`"

            # LETS CHECK THE RESULT
            if [[ $PING1 =~ "OK" ]]
            then
                # IF THE RESULT WAS OK THEN LOG THE RESULT
                echo $(date) OK ${IPS[$i]} ${HOSTS[$i]} $PING1 >> $LOG
            else
                # IF THE RESULT WAS BAD THEN LOG AND EMAIL AND COLLECT A TRACEROUTE
                echo "FAIL RETRY PING ${IPS[$i]} ${HOSTS[$i}} $PING1" | $SENDMAIL -s "$MAIL_SUBJECT" $MAIL_TO
                echo $(date) FAIL ${IPS[$i]} ${HOSTS[$i]} $PING1 >> $LOG
                `${MTR} --no-dns -rc 5 ${IPS[$i]} >> $LOG`
            fi
        fi
    fi
done

# LETS WAIT FOR A FEW SECONDS
sleep 10

# LETS CLEAN UP AND REMOVE THE LOCKFILE
rm -f ${LOCKFILE}

# THATS ALL FOLKS!
exit 0

Cheers!

Note: This is a series of posts made under the Network Engineer in Retail 30 Days of Peak, this is post number 29 of 30. All the posts can be viewed from the 30in30 tag.

Image Credit: michele de notaristefani

We utilize BGP to provide dynamic routing between the many Internet Service Providers we are peering with and at the many Data Centers and circuits over which we peer. This past weekend we had an issue with our primary Internet Service Provider (AT&T) but BGP did it’s job and dutifully detected the dead router and re-routed traffic to the remaining Internet Service Providers. The actual outage time was less then 60 seconds. Even though it occurred around 1:30AM EST in the morning we’re hosting websites that need to be accessible in every timezone around the world. While it was 1:30AM on the East coast it was only 10:30PM on the West coast where shoppers were still busy picking through the online goods and placing orders. And while it might have been a little too early for our friends in the UK (6:30AM GMT), we could have shoppers online from either France or Germany (7:30AM GMT +1).

Dec 14 2014 01:27:20.337: %BGP-5-ADJCHANGE: neighbor 12.251.xxx.xxx Down BGP Notification sent
Dec 14 2014 01:27:20.337: %BGP-3-NOTIFICATION: sent to neighbor 12.251.xxx.xxx 4/0 (hold time expired) 0 bytes
Dec 14 2014 01:27:22.650: %BGP_SESSION-5-ADJCHANGE: neighbor 12.251.xxx.xxx IPv4 Unicast topology base removed from session  BGP Notification sent
Dec 14 2014 01:33:25.052: %BGP-5-ADJCHANGE: neighbor 12.251.xxx.xxx Up

We also utilize BGP internally in combination with BFD (Bidirectional Forwarding Detection) to help reduce the failover time on the internal network. We’ve actually had BFD accidentally trip a number of times because it can be too sensitive which can create just as many issues having routes flapping back and forth between multiple paths.

As of this writing I have ~ 511,000 IP routes in my BGP routing tables.

Looking at a peering point on the East coast of the United States;

511499 network entries using 132989740 bytes of memory
2550193 path entries using 244818528 bytes of memory
836860/82048 BGP path/bestpath attribute entries using 187456640 bytes of memory
293120 BGP AS-PATH entries using 13504896 bytes of memory
12459 BGP community entries using 1489256 bytes of memory
51 BGP route-map cache entries using 3264 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 580262324 total bytes of memory
BGP activity 11331307/10819807 prefixes, 150321731/147771538 paths, scan interval 60 secs

Here’s a look at a peering point on the West coast of the United States;

511029 network entries using 132867540 bytes of memory
1021218 path entries using 98036928 bytes of memory
246998/81716 BGP path/bestpath attribute entries using 55327552 bytes of memory
145392 BGP AS-PATH entries using 6562258 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 292794278 total bytes of memory
BGP activity 4607568/4096503 prefixes, 24245483/23224265 paths, scan interval 60 secs

The delta in path entries between the two is a result of the number of BGP peers I have on that specific router.

East = 2,550,193
West = 1,021,218

As you can guess I have a number of additional peers on the East coast than I have on the West coast – plans are in the works to resolve that next calendar year.

You can see the dramatic growth in the number of BGP routes being advertised over the Internet from http://bgp.potaroo.net/.

Cheers!

Note: This is a series of posts made under the Network Engineer in Retail 30 Days of Peak, this is post number 21 of 30. All the posts can be viewed from the 30in30 tag.

Articles

SPARK: VIRL is launched! by Anthony Burke – I’m curious to see what VIRL does to the GNS3 community, especially with GNS3 having just recently released their new version, GNS3 1.2. The hardware requirements for VIRL are pretty stiff compared with GNS3, and I also noticed there’s no support for VirtualBox. I personally haven’t had time to play around with GNS3 but hope to-do so sometime soon. I’m curious to read what others are thinking.

Using Scapple To Help Manage Complex Network Changes by Ethan Banks – I enjoy using Scapple to build flow charts for data or interfaces. You’d be surprised how many firewalls, load-balancers and reverse proxy servers some data feeds need to traverse. Scapple is a great tool, quick and dirty, that allows me to quickly document the existing environment and review new designs. I’ve also used Skitch which was acquired by Evernote sometime ago.

The Android 5.0 Lollipop Review by Brandon Chester – I’m still running Android 4.4.2 (Kitkat) but I’m looking forward to testing Lollipop when either Verizon releases it for Moto X or Samsung releases it for their Galaxy Note 10.1 2014 Edition tablet.

Data caps, limited competition a recipe for trouble in home Internet service by Jon Brodkin – The war around Net Neutrality is heating up again. I’m a Verizon FiOS TV/Internet customer and they don’t have data caps yet, but I can’t see it staying that way for long if Comcast is allowed to continue down the path their on. I really wish we had more competition for broadband here in the United States.

Closing Comments on Old Posts by Scott Lowe – I threw in the towel earlier this year and now close comments on posts older than a year. Since that change the comment SPAM on my blog has dropped off significantly. I now get 40-60 a week where as I was getting 200-350 a week.

Can You AS-Prepend a Single Host Route? by Ivan Pepelnjak – I’ve used AS prepending on /32 (host) routes internally within my network to provide redundant paths to critical VIPs and such. As mentioned by Ivan I don’t think there are any ISPs out there that will accept anything less than a /24 route into their BGP routing tables.

Cheers!

Note: This is a series of posts made under the Network Engineer in Retail 30 Days of Peak, this is post number 10 of 30. All the posts can be viewed from the 30in30 tag.

Once we made the BGP local-pref change all IP traffic was egressing the AT&T circuit as visualized below.

There was never a problem reaching any of the ARIN IP address blocks that we were advertising via BGP the problem was isolated to just the WAN IP interfaces of the other Internet Service Providers.

The problem turned out to be that traffic to the WAN IP address was ingressing the circuit that the IP address was assigned to but it would egress the AT&T circuit due to the BGP local-pref statement. I’m guessing that AT&T is filtering the traffic on ingress checking for traffic sourced from an IP address block that has no business coming from that link and was dropping the traffic.

So an ICMP packet to the Comcast WAN IP address would ingress the Comcast interface and would egress the AT&T interface with a source IP address of the Comcast WAN interface. That packet would hit the AT&T head-end router which would discard any packets not sourced from the a valid ARIN IP address block belonging to that link, similar to Reverse Path Forwarding. I was able to verify this by placing a pair of static routes on the router using the Comcast circuit as a return path and with that the two Linux servers were now able to ping all the WAN IP interfaces. I’m guessing that while AT&T does some source route filtering, Comcast isn’t doing any.

It think it’s great that AT&T is filtering their inbound traffic for valid source IP blocks, it definitely helps prevent IP spoofing.

The confusion came when I did a debug ip icmp and later a debug ip packet 100 detail and observed no ICMP traffic coming from either of the two Linux servers on the Cisco ASR1001. I had a ticket open with Cisco TAC and they were also unable to explain the oddity. I’m curious if this was something to-do with CEF and might I need to enable no ip route-cache on the specific interfaces?

Cheers!

It’s fascinating now that I’m working in retail to follow the actual process flow from order entry to order fulfillment.

In any case a few quick tests using ICMP pings didn’t show any issues or problems. However, a subsequent packet trace performed from the server revealed a large number of TCP Re-transmissions and Duplicate ACKs. It was pretty clear to me that we had some significant packet loss between the two servers. However the vendor felt it was indicative of “application packet loss“. I’ve been in the networking field for quite a few years now… I’ve seen a lot and heard a lot but I’ve never heard the phrase “application packet loss”. The vendor was suggesting that it was the application that was causing the TCP Re-transmissions and Duplicate ACKs and that the network was not to blame.

In classic fashion I politely called bullshit, ok maybe I was a little more forceful than that.

It was the TCP Re-transmissions that was causing the slow down in the transaction exchanges. The packets were being re-transmitted because they were being lost somewhere between the two servers. I could see a 8 second delay here, a 8 second delay there… when you add them up you get interfaces that generally take 200ms to exchange data taking upwards of 20-40 seconds. The larger problem, we had 5000+ transactions backing up and we were falling further and further behind since the rate at which the transactions were entering the queue was far outpacing the rate at which the transactions were being processed.

In the end the vendor changed some of their Internet BGP peering in order to leverage a different Internet provider and path and that magically solved the problem instantly. There was some peering point out on the Internet that was throwing out some packets and that was causing our issues.

If you’ve ever heard of application packet loss please by all means please educate me!

Cheers!

References;
Why does packet loss destroy application performance over the WAN? by Andy Gottlieb

Before the BGP Soft Reset Enhancement feature, a soft reset for inbound routing table updates was performed by entering the neighbor soft-reconfiguration router configuration command. This command was used to configure the local BGP router to store all received (inbound) routing policy updates. However, this method uses too much memory because inbound updates are not modified and is not recommended.

I’m guessing this new feature had a significant impact for anyone taking a full Internet BGP table?

Cheers!

Image Credit: Earth 3D by Jan

Layer 3 and vPC Recommendations

I was provided a copy of the Nexus 7000 virtual Port-Channel Best Practices & Design Guidelines which clearly indicates on page 25 that routers should not be connected to a vPC link but should instead be connected via a Layer 3 switch port. Here are some bullet points;

• Use separate L3 links to hook up routers to a vPC domain is still standing.
• Don’t use L2 port channel to attach routers to a vPC domain unless you can statically route to HSRP address
• If both, routed and bridged traffic is required, use individual L3 links for routed traffic and L2 port-channel for bridged traffic

I was still currious to understand more of the inner-workings.. why didn’t it work or wasn’t it allowed? I only had to flip through the next few slides although I can really say that I completely understand just yet.

1. Packet arrives at R
2. R does lookup in routing table and sees 2 equal paths going north (to 7k1 & 7k2)
3. Assume it chooses 7k1 (ECMP decision)
4. R now has rewrite information to which router it needs to go (router MAC 7k1 or 7k2)
5. L2 lookup happens and outgoing interface is port-channel 1
6. Hashing determines which port-channel member is chosen (say to 7k2)
7. Packet is sent to 7k2
8. 7k2 sees that it needs to send it over the peer-link to 7k1 based on MAC address
9. 7k1 performs lookup and sees that it needs to send to S
10. 7k1 performs check if the frame came over peer link & is going out on a vPC.
11. Frame will only be forwarded if outgoing interface is NOT a vPC or if outgoing vPC doesn’t have active interface on other vPC peer (in our example 7k2)

I’m not embarrassed to say that I followed everything up until step 11. Why exactly is it that frames will only be forwarded if the outgoing interface is NOT a vPC or if the outgoing vPC doesn’t have an active interface on another vPC peer? Isthere anyone that can shed any additional light on this topic?

I’ve never experienced such a restriction in all my years of working with the Avaya (formerly Nortel) Ethernet Routing Switch 8600 and their Split Multilink Trunking (SMLT) technology. I actually have a Cisco 3825 router connected via a SMLT attached Ethernet Routing Switch 5520 (Layer 2) with the Cisco 3825 and the Avaya 8600s all running BGP.

Cheers!

If you are looking for a great resource on BGP I would highly recommend O’Reilly’s book titled BGP.

If you are looking for Avaya/Nortel specific information concerning their BGP implement then you are in luck. Avaya has a technical configuration guide for the ERS 8600 that focuses on BGP. While this is an older document (November 2007) it still does a great job of providing a number of configuration examples and explaining the basics.

In the near future I might need to use an ERS 8606 as an Internet router. I’ll need to peer with the ISP since I’m multi-home to independent Internet Service Providers, although I’m not sure if the 8692SF can handle a full BGP routing table. Has anyone ever tried to feeding a full (or partial) BGP routing table from the Internet to an ERS 8800/8600 switch?

Cheers!