Michael McNamara

technology, networking, virtualization and IP telephony

Cisco ASA Firewall breaks after 213 days of uptime

by

I just recently had two HA pairs of Cisco ASA firewalls just stop communicating. A reboot of both the primary and secondary firewall in each HA pair resolved the problem. I had never observed such odd behavior from two pairs of Cisco ASA firewalls so I immediately suspected either a possible public exploit or a software bug given that both HA pairs were upgraded within the past 6-7 months.

Upon reviewing the 9.1.7 release notes from Cisco I stumbled over the following entry;

CSCvd78303 – ARP functions fail after 213 days of uptime, drop with error ‘punt-rate-limit-exceeded’

We upgraded a large number of our Cisco ASAs about 213 days ago to 9.1.(7)12 and now it would seem we’ll be upgrading again to 9.1.(7)19 – the actual issue is resolved in 9.1(7)16.

Here’s the text of the bug report;

Symptom:
An ASA, after reaching an uptime of roughly 213 days will fail to process ARP packets leading to a condition where all traffic eventually stops passing through the affected device. Since not all existing ARP entries time out at the same time, not all connections may fail at the same time.
Additional symptoms include:

  • ASA does not have ARP entries in its ARP table. show arp is empty
  • The output of show asp drop and ASP drop captures indicate a rapidly increasing counter for punt-rate-limit exceeded and the dropped packets are predominantly ARP.

IMAGES WITH FIXES
Images with fixes for this defect will be published as soon as they are available, and posted to the Cisco Software Download center.
Conditions:
This is seen when the ASA’s uptime reaches 213 days.
This problem affects ASA and FTD versions:
ASA version 9.1 releases 9.1(7)8 and higher
ASA version 9.2 releases 9.2(4)15 and higher
ASA version 9.4 releases 9.4(3)5 and higher including 9.4(4)
ASA version 9.5 releases 9.5(3) and higher
ASA version 9.6 releases 9.6(2)1 and higher including 9.6(3)
ASA version 9.7 releases 9.7(1) and higher
FTD version 6.1 releases 6.1.0.1 and higher
FTD version 6.2 releases 6.2.0

Workaround:
Perform a pre-planned reboot of the device before approaching the 213 days (5124 hours) of up time. After the reboot, it will give you another 213 days of up time.
Further Problem Description:
Devices encountering this issue will not receive or respond to ARP packets. This affects not just transient traffic, but also access to the affected device – including Administration access such as SSH, HTTPS and Telnet. Console access is not affected.

If your running a Cisco ASA firewall I would recommend you check to make sure that you won’t be impacted by this bug.

Cheers!

Network is down! Please help!

by

I get quite a few emails and and private messages that contain the phrase above, “Network is down! Please help!”. In the vast majority of cases it usually turns out that the problem is self inflicted. Here are a few simple steps that I strongly recommend you look at if it’s your responsibility to manage the network. These features will help mitigate some serious operational issues from spoiling your day.

Spanning Tree (FastStart / PortFast)

You can’t imagine how many times I’ve seen a patch cable patched from one port on a switch in an IDF back into the same switch. It’s usually a case of sloppy cabling, although I’ve met a quite a few users who thought that both ends of the cable needed to be plugged into the wall plate. In order to prevent such scenarios from spoiling your day you should verify that you have Spanning Tree enabled on all edge ports (ports that face the users). And you should also verify that you have FastStart or PortFast enabled to help eliminate the 30 second delay during STP learning (those of us that used to run IPX/SPX networks with Novell NetWare learned this a long time ago).

BPDU Guard (called BPDU Filtering by some vendors)

In addition to Spanning Tree you should also enable BPDU guard on all your edge ports. This will immediately disable the switchport if it receives a BPDU Spanning Tree frame. This can help mitigate topology loops and it can help prevent your users from plugging in their own switches in their offices or cubicles. There are some scenarios where connecting the PC port on the back of an Avaya IP phone could potentially put a loop into the network. BPDU guard (called BPDU filtering by Avaya) will detect the BPDU frame and disable the port, thereby protecting the network.

Rate Limiting (Broadcast / Multicast)

If your in the networking field long enough you’ll eventually see some really odd behavior from one or more devices. I’ve seen my fair share of oddities and eventually you’ll run across a few too. You should prepare for that eventuality today by configuring broadcast and Multicast rate limiting on your edge ports. This helps prevent any single device from flooding too many broadcast or Multicast packets into your network which could impact either other devices connected to the network or the network itself.

You’re already doing each of the previously listed items? Well you could look at the following additional steps. You can find additional detail in a blog post titled, DHCP Snooping ARP Inspection IP Source Guard.

DHCP Snooping

Dynamic Host Configuration Protocol (DHCP) snooping provides security to the network by preventing DHCP spoofing. DHCP spoofing refers to an attacker’s ability to respond to DHCP requests with false IP information. DHCP snooping acts like a firewall between untrusted hosts and the DHCP servers, so that DHCP spoofing cannot occur. There are 2 types of ports in DHCP snooping, trusted and untrusted. The network will only allow DHCP responses from trusted ports – usually uplinks as opposed to allowing DHCP responses from untrusted ports – usually edge switch ports.

DHCP Option 82 – With DHCP Option 82 the switch will append additional information to the DHCP request which can be stored and captured by your DHCP server (or your IP Address Management solution) to help identify the switch and port from which the request was initiated.

ARP Inspection

Dynamic Address Resolution Protocol Inspection (DAI) is a security feature that validates ARP packets in the network.

Without dynamic ARP inspection, a malicious user can attack hosts, switches, and routers connected to the Layer 2 network by poisoning the ARP caches of systems connected to the subnet and by intercepting traffic intended for other hosts on the subnet. Dynamic ARP inspection prevents this type of attack. It intercepts, logs, and discards ARP packets with invalid IP-to-MAC address bindings.

The address binding table is dynamically built from information gathered in the DHCP request and reply when DHCP snooping is enabled. The MAC address from the DHCP request is paired with the IP address from the DHCP reply to create an entry in the DHCP binding table.

When you enable Dynamic ARP inspection, ARP packets on untrusted ports are filtered based on the source MAC and IP addresses stored in the DHCP snooping table. The switch forwards an ARP packet when the source MAC and IP address matches an entry in the DHCP snooping table. Otherwise, the ARP packet is dropped.

IP Source Guard

IP Source Guard provides security to the network by filtering clients with invalid or spoofed IP addresses. IP Source Guard is a Layer 2 (L2), port-to-port feature that works closely with information in the Dynamic Host Control Protocol (DHCP) snooping binding table. When you enable IP Source Guard on an untrusted port with DHCP snooping enabled, an IP filter entry is created or deleted for that port automatically, based on IP information stored in the corresponding DHCP binding table entry. When a connecting client receives a valid IP address from the DHCP server, a filter is installed on the port to allow traffic only from the assigned IP address. A maximum of 10 IP addresses are allowed on each IP Source Guard-enabled port. When this number is reached, no more filters are set up and traffic is dropped.

While Dynamic ARP Inspection blocks only ARP packets, IP Source Guard blocks all IP packets.

Cheers!
Image Credit: The Lánchid Bridge in Budapest Hungary at nightfall by Abel Leemans

DHCP Snooping ARP Inspection IP Source Guard

by

890258_17394208Here are three features that are often not implemented in most networks but could help to really address security shortcomings and potential operational issues.

I’ve recently started implementing DHCP Snooping, Dynamic ARP Inspection and IP Source Guard in my networks so I thought I’d take the time to strike up a conversation around those features and hear what everyone else is doing.

These features are all intended to provide a more stable network environment and help protect against intentional and unintentional events that can interfere with the proper operation of the network. Before we can implement any of these features we need to understand how they work and what changes (if any) will be needed to support them.

Hopefully most people have Spanning Tree, FastStart, BPDU filtering (guard) and Broadcast/Multicast rate limiting enabled by now but if you don’t you should certainly look at enabling and configuring those features before you start to try any of the ones I’m about to discuss.

The sample configuration code provided below is applicable for Avaya Ethernet Routing Switches.

DHCP Snooping

Dynamic Host Configuration Protocol (DHCP) snooping provides security to the network by preventing DHCP spoofing. DHCP spoofing refers to an attacker’s ability to respond to DHCP requests with false IP information. DHCP snooping acts like a firewall between untrusted hosts and the DHCP servers, so that DHCP spoofing cannot occur. There are 2 types of ports in DHCP snooping, trusted and untrusted. The network will only allow DHCP responses from trusted ports – usually uplinks as opposed to allowing DHCP responses from untrusted ports – usually edge switch ports.

Example, if someone brings a Linksys router into the office and connects the LAN side to your network the Linksys router will start handing out 192.168.1.x IP DHCP addresses which will likely cause some chaos. DHCP Snooping will prevent those DHCP responses from poisoning your network. I’m happy to admit that this is happened to me on more than one occasion, usually at our remote finance office where auditors think their second job could be in Information Technology.

DHCP Option 82 – With DHCP Option 82 the switch will append additional information to the DHCP request which can be stored in your IPAM (IP Address Management) solution to help identify the switch and port from which the request was initiated. This can provide diagnostic and troubleshooting information which can all be stored directly in your IPAM if it supports DHCP Option 82. Infoblox supports DHCP Option 82.

The latest software releases support the ability to copy the DHCP binding table up to a TFTP server. This prevents the switch from throwing away all the previously learned DHCP transactions between reboots or restarts of the switch.

In this example our uplinks are 1/24 and 2/24, all interfaces are untrusted by default.

ip dhcp-snooping vlan 10
ip dhcp-snooping vlan 11
ip dhcp-snooping enable

interface fa 1/24,2/24
 ip dhcp-snooping trusted
exit

If you want to enable DHCP Option 82 support;

ip dhcp-relay option82
interface vlan 10
 ip dhcp-relay option82
interface vlan 11
 ip dhcp-relay option82

You can also edit the text string the switch will append per port with the following;

interface FastEthernet 1
 ip dhcp-relay option82-subscriber-id <string 1..255>

Issues with DHCP Snooping

While you need to properly identify trusted and untrusted ports there are very few draw backs to utilizing DHCP snooping. It only prevents DHCP replies from untrusted ports essentially preventing DHCP spoofing in the network. DHCP Snooping can be enabled in most networks with a few configuration changes and very little danger (impact).

Dynamic ARP Inspection

Dynamic Address Resolution Protocol (Dynamic ARP) inspection is a security feature that validates ARP packets in the network.

Without dynamic ARP inspection, a malicious user can attack hosts, switches, and routers connected to the Layer 2 network by poisoning the ARP caches of systems connected to the subnet and by intercepting traffic intended for other hosts on the subnet. Dynamic ARP inspection prevents this type of attack. It intercepts, logs, and discards ARP packets with invalid IP-to-MAC address bindings.

The address binding table is dynamically built from information gathered in the DHCP request and reply when DHCP snooping is enabled. The MAC address from the DHCP request is paired with the IP address from the DHCP reply to create an entry in the DHCP binding table.

When you enable Dynamic ARP inspection, ARP packets on untrusted ports are filtered based on the source MAC and IP addresses stored in the DHCP snooping table. The switch forwards an ARP packet when the source MAC and IP address matches an entry in the DHCP snooping table. Otherwise, the ARP packet is dropped.

For dynamic ARP inspection to function, DHCP snooping must be globally enabled. Dynamic ARP inspection is configured on a VLAN to VLAN basis.

Let’s enable Dynamic ARP Inspection on VLANs 10, 11 and set our uplinks as trusted;

ip arp-inspection vlan 10
ip arp-inspection vlan 11
ip arp-inspection enable

interface fa 1/24,2/24
 ip arp-inspection trusted
exit
interface fa 1/1-23,2/1-23
 ip arp-inspection untrusted
exit

Issues with Dynamic ARP Inspection

Dynamic ARP Inspection relies on the information stored in the DHCP binding table (from DHCP Snooping) to validate the ARP packets it receives on untrusted ports. Any device whether it be statically configured or dynamically configured would need to appear in the DHCP binding table.

If you have a statically configured device you’ll need to manually populate the DHCP snooping table. If someone changed out a statically configured device the MAC address would most likely need to be updated in the DHCP binding table. If the DHCP binding table was accidentally cleared the switch would block IP traffic until the DHCP binding table was re-built either manually or from DHCP transactions.

This is another great reason to use manual or reserved DHCP assignments where possible if the device requires a persistent IP address.

This feature will likely create some significant administrative overhead based on the number of devices configured with a static IP address over the number of DHCP configured devices.

IP Source Guard

IP Source Guard provides security to the network by filtering clients with invalid or spoofed IP addresses. IP Source Guard is a Layer 2 (L2), port-to-port feature that works closely with information in the Dynamic Host Control Protocol (DHCP) snooping binding table. When you enable IP Source Guard on an untrusted port with DHCP snooping enabled, an IP filter entry is created or deleted for that port automatically, based on IP information stored in the corresponding DHCP binding table entry. When a connecting client receives a valid IP address from the DHCP server, a filter is installed on the port to allow traffic only from the assigned IP address. A maximum of 10 IP addresses are allowed on each IP Source Guard-enabled port. When this number is reached, no more filters are set up and traffic is dropped.

While Dynamic ARP Inspection blocks only ARP packets, IP Source Guard blocks all IP packets.

interface fa 1/1-23,2/1-23
 ip verify source
exit

Issues with IP Source Guard

IP Source Guard utilizes the information stored in the DHCP binding table (from DHCP Snooping) to validate the IP traffic. Any device whether it be statically configured or dynamically configured would need to appear in the DHCP binding table. Statically configured devices would need to be manually placed in the DHCP binding table. If someone changed out a device the MAC address would most likely need to be updated in the DHCP binding table. If the DHCP binding table was accidentally cleared the switch would block IP traffic until the DHCP binding table was re-built either manually or from DHCP transactions.

This feature will likely create some significant administrative overhead based on the number of devices configured with a static IP address over the number of DHCP configured devices.

My Thoughts

I’m taking a split approach right now, let me explain. I’ll be utilizing DHCP snooping and DHCP Option 82 throughout my network. Unfortunately due to the administrative overhead required to leverage Dynamic ARP Inspection and IP Source Guard I’m only planning on implementing those features at my smaller remote offices – the ones I usually have issues with from a security perspective. Since those remote offices are much smaller and usually don’t make too many changes it offers me the opportunity to evaluate the new features without over taxing my network administrators and engineers.

Cheers!

Perl Script to poll ARP Table

by

I’ve written a lot of Perl scripts to help make managing the network easier and more efficient. One of the scripts I’ve written allows me to dump the IP ARP table of the Nortel Ethernet Routing Switch 8600 to a file for later/additional processing. While the script was original written for the ERS 8600 switch it will also work on just about any router (Layer 3 device) that supports the RFC1213 (ipNetToMediaNetAddress).

The script has been tested and works on Nortel’s BayRS routers (ARN, ASN, BLN, BCN). You just obviously need to be careful of how the script interprets the ipNetToMediaIfIndex value depending on the device you are polling.

The script get8600arp.pl is a very straight forward script. It simply polls various SNMP OIDs and then stores the results in a file. It does this for every switch (FQDN/IP Address) that is listed in the input file.

#!/usr/bin/perl
#
# Filename: /root/get8600arp.pl
#
# Purpose:  Query Nortel Ethernet Routing Switch 8600 for the IP ARP
#           table via SNMP. This script will poll a list of devices
#           (input file) and dump the contents of the IP ARP table to
#           and outputfile.
#
# Author:   Michael McNamara
#
# Date:     December 5, 2002
#
# Support Switches:
#           - Nortel ERS 8600
#           - Nortel ERS 1600
#           - Nortel ERS 5500
#           - Nortel BayRS Routers
#
# Requirements:
#           - Net-SNMP
#           - Net-SNMP Perl Module
#           - SNMP-MIBS
#
# Changes:
#
#           - May  5, 2007 (M.McNamara)
#           clean up code and documentation for release to public
#           - Oct 10, 2006 (M.McNamara)
#           went back to SNMP v1 to support BayRS legacy routers
#           - Sep 04, 2003 (M.McNamara)
#           migrated from vendor specific MIB to RFC1213 (ipNetToMediaNetAddress)
#

# Load Modules
use strict;
use SNMP;
use Net::Ping;

# Declare constants
#use constant DEBUG      => 0;           # DEBUG settings
use constant RETRIES    => 3;           # SNMP retries
use constant TIMEOUT    => 1000000;     # SNMP timeout, in microseconds
use constant SNMPVER    => 1;           # SNMP version

# SNMP Settings
$SNMP::verbose = 0;
$SNMP::use_enums = 1;
$SNMP::use_sprint_value = 0;
&SNMP::initMib();
&SNMP::loadModules('RAPID-CITY');

# Declaration Variables
my ($sess, @vals);
my @devices;
my ($card, $port);
my $snmphost;
my $comm = "public";        # SNMP ReadOnly Community String
my %array;
my $switchfile;
my $datafile;

our $DEBUG;                     # DEBUG flag

undef @devices;

# Program and help information
my $program = "get8600arp.pl";
my $version = "v1.3";
my $author = "Michael McNamara";
my $purpose = "This Perl script is retreieve the IP ARP table from the ERS8600 Layer 3 switch/router and store it in file for later use.";
my $usage = "Usage: $program \[input\] \[output\] \[-help\] \[debug\]\n    <input>  = filename listing each switch to poll\n    <output> = filename where to store output\n";

if (($#ARGV +1) <= 2) {
 print "Program: $program \nVersion: $version \nWritten by: $author \n$purpose\n\n$usage\n";
 print "DEBUG: ARGV =  $#ARGV\n";
 print "DEBUG: ARGV =  $ARGV[0] $ARGV[1] $ARGV[2] $ARGV[3]\n";
 exit;
}

my $arg1 = shift @ARGV;
my $arg2 = shift @ARGV;
my $arg3 = shift @ARGV;

if ($arg1 =~ /help/) {
 print "Program: $program \nVersion: $version \nWritten by: $author \n$purpose\n\n$usage\n";
 print "DEBUG: ARGV =  @ARGV\n";
 print "DEBUG: ARGV =  $ARGV[0] $ARGV[1] $ARGV[2] $ARGV[3]\n";
 exit;
}

$switchfile = $arg1;
$datafile = $arg2;
$DEBUG = $arg3;

# Test to see if inputifle exists
if (!-e $switchfile) {
 die "ERROR: Unable to locate and/or open inputfile $switchfile...";
}

############################################################################
##### B E G I N   M A I N ##################################################
############################################################################

&load_switches;

&collect_arp;

exit 0;

############################################################################
#### E N D   M A I N #######################################################
############################################################################

############################################################################
# Subroutine collect_arp
#
# Purpose: collect ARP information from layer 3 switches/routers
############################################################################
sub collect_arp {

 # Open output datafile for appending
 open(DATAFILE, ">>$datafile");

 # Loop over each Passport 8600 switch
 foreach $snmphost (@devices) {

    my $packet = Net::Ping->new('icmp');

    $snmphost =~ s/\n//g;        # remove CRLF

    if ($packet->ping($snmphost)) {

       $sess = new SNMP::Session (    DestHost   =>  $snmphost,
                              Community  =>  $comm,
                              Retry      =>  RETRIES,
                              Timeout    =>  TIMEOUT,
                              Version    =>  SNMPVER );

       my $vars = new SNMP::VarList(
                              ['ipNetToMediaIfIndex', 0],
                              ['ipNetToMediaPhysAddress', 0],
                              ['ipNetToMediaNetAddress', 0],
                              ['ipNetToMediaType', 0] );

       while (1) {

          @vals = $sess->getnext($vars);  # retreive SNMP information

          last unless ($vars->[0]->tag eq 'ipNetToMediaIfIndex');

          $vals[1] = unpack('H12', $vals[1]);
          $vals[1] =~ tr/a-z/A-Z/;

          $card = (($vals[0] & 62914560) / 4194304);
          $port = (($vals[0] & 4128768) / 65536) + 1;

          print "$snmphost, $vals[0], ($card/$port), $vals[1], $vals[2], $vals[3]\n" if ($DEBUG);
          print DATAFILE "$snmphost, $vals[0], $card, $port, $vals[1], $vals[2]\n";

          $array{$snmphost}[$card][$port] = $vals[2];

       } # end while

    } else {

       print ("ERROR: $snmphost not responding to ICMP ping skipping...\n");

    } #end if $packet

 } #end foreach

 close(DATAFILE);

} #end sub collect_arp

############################################################################
# Subroutine load_switches
#
# Purpose: load list of switches
############################################################################
sub load_switches {

 open(SWITCHLIST, "<$switchfile");

 # Walk through data file
 while (<SWITCHLIST>) {

    # Skip blank lines
    next if (/^\n$/);
    # Skip comments
    next if (/^#/);

    #print "DEBUG: adding $_ to our list of devices \n" if ($DEBUG);

    push (@devices, $_);

 }

 close(SWITCHLIST);

 return 1;

} # end sub load_switches
############################################################################

The real magic that folks have always been searching for is the binary formula to turn the ipNetToMediaIfIndex into a location that denotes the card and port where that specific device is connected to.

$card = (($vals[0] & 62914560) / 4194304);
$port = (($vals[0] & 4128768) / 65536) + 1;

While I still use flat files you could certainly adopt this code to dump the output into a database. I just haven’t had the time although I’ve been playing with MySQL quite a bit lately.

Cheers!

What are the ARP and FDB tables?

by

I’ll try to describe and explain the purpose behind the ARP and FDB tables in networking. I will be the first to admit that there are probably much better descriptions that can be found elsewhere on the net.

The ARP (Address Resolution Protocol) table is used by a Layer 3 device (router, switch, server, desktop) to store the IP address to MAC address entries for a specific network device. The ARP table allows a device to resolve a Layer 3 address (IP address) into a Layer 2 address (MAC address). The ARP table is populated as devices issue ARP broadcasts looking for a network device’s Layer 2 (MAC address).

How does it work? When a Layer 3 device has an IP packet that it needs to deliver to a locally attached interface it will look to the ARP table to figure out what MAC address to put into the packet header. The important point above is “a locally attached interface”. If the IP packet is destined for a remote network it will be routed per the routing table. If there is no ARP table entry for the destination IP address the Layer 3 device will try ARP broadcasting for it. Once it has the MAC address for that specific IP address it will forward the packet with the appropriate MAC address in headers. Example; you can list the ARP table of a Windows XP computer by using the following command at the DOS prompt, “arp -a”.

The FDB (forwarding database) table is used by a Layer 2 device (switch/bridge) to store the MAC addresses that have been learned and which ports that MAC address was learned on. The MAC addresses are learned through transparent bridging on switches and dedicated bridges.

How does it work? When a Ethernet frame arrives at a Layer 2 device, the Layer 2 device will inspect the destination MAC address of the frame and look to its FDB table for information on where to send that specific Ethernet frame. If the FDB table doesn’t have any information on that specific MAC address it will flood the Ethernet frame out to all ports in the broadcast domain.

A Layer 3 switch performs both the routing and switching in a single device. It will typically have both an ARP and FDB table and it will perform both tasks depending on whether the packet/frame needs to be routed or switched. The Nortel Ethernet Routing Switch 8600 is a Layer 3 switch while the Nortel Ethernet Switch 470 is a Layer 2 switch. The Nortel Ethernet Routing Switch 5500 Series is also a Layer 3 device that can be used a Layer 2 device if desired.

Let me point out that Wikipedia is a great resource these days for an amazing number of topics. It’s a world-wide collaborative effort with over 75,000 contributors. Anyone can sign-up and contribute content in whatever subject material they are knowledgeable in. It’s probably best described as the world’s largest growing online encyclopedia.

Have a look at the following Wikipedia entry;

http://en.wikipedia.org/wiki/Ethernet

There is an amazing amount of information in those articles with an equally amazing amount of detail. Thanks to everyone who contributes to Wikipedia!

Cheers!

Recent Posts

Categories

SUBSCRIBE BY EMAIL