The application was the original suspect, and since I wrote it years back I was asked to look at the problem. Whenever we add a new model of switch, be it a Cisco Nexus 2248TP or Avaya ERS 4850-GTS-PWR+  there’s usually some tweaking involved to make sure that everything works properly. That’s the price you pay by writing your own software solutions. This time around however it became clear pretty quickly that something else was wrong. Initially I was puzzled since every snmpwalk I performed on the ERS 4850 returned the proper values. It wasn’t until I crafted a command line with multiple SNMP OIDs (just like the script) that I was able to observe the problem.

The problem appears to be related to how the Avaya ERS 4850-GTS-PWR+ handles SNMP queries with multiple SNMP OIDS included in the same request. If I perform a SNMP query for each of the following OIDs in the same request I get the same incorrect ifInOctets value back for each port.

• 1.3.6.1.2.1.2.2.1.1.38 – ifIndex
• 1.3.6.1.2.1.2.2.1.10.38 – ifInOctets
• 1.3.6.1.2.1.2.2.1.3.38 – ifType

Notice how the value is the same for every port, although if I re-query the switch it will provide a different value for every port. In short the incorrect value breaks the application since it appears that every port is changing daily and no ports are ever becoming idle.

root@roo ~]# snmpgetnext -v2c -cpublic sw-icr3-psyc.acme.org ifIndex.1 ifInOctets.1 ifType.1
IF-MIB::ifIndex.2 = INTEGER: 2
IF-MIB::ifInOctets.2 = Counter32: 1106547808
IF-MIB::ifType.2 = INTEGER: ethernetCsmacd(6)

[root@roo ~]# snmpgetnext -v2c -cpublic sw-icr3-psyc.acme.org ifIndex.2 ifInOctets.2 ifType.2
IF-MIB::ifIndex.3 = INTEGER: 3
IF-MIB::ifInOctets.3 = Counter32: 1106547808
IF-MIB::ifType.3 = INTEGER: ethernetCsmacd(6)

[root@roo ~]# snmpgetnext -v2c -cpublic sw-icr3-psyc.acme.org ifIndex.3 ifInOctets.3 ifType.3
IF-MIB::ifIndex.4 = INTEGER: 4
IF-MIB::ifInOctets.4 = Counter32: 1106547808
IF-MIB::ifType.4 = INTEGER: ethernetCsmacd(6)

If I issue a SNMP get next for just the single OID then the switch returns the correct value;

[root@roo ~]# snmpgetnext -v2c -cpublic sw-icr3-psyc.acme.org ifInOctets.1  
IF-MIB::ifInOctets.2 = Counter32: 3903266154

[root@roo ~]# snmpgetnext -v2c -cpublic sw-icr3-psyc.acme.org ifInOctets.2
IF-MIB::ifInOctets.3 = Counter32: 2492668434

[root@roo ~]# snmpgetnext -v2c -cpublic sw-icr3-psyc.acme.org ifInOctets.3
IF-MIB::ifInOctets.4 = Counter32: 792830238

The result is the same whether I use SNMP v1 or SNMP v2c.

The script itself really isn’t concerned with precision, we actually only record the last 6 digits of the counter. If we were concerned about precision we might have to start utilizing ifHCInOctets (1.3.6.1.2.1.31.1.1.1.6) since this is a 10/100/1000Mbps switch port and the counters might wrap between polls.

I’ve only seen the problem on the Avaya ERS 4850-GTS-PWR+ switch running HW:10 FW:5.6.2.1   SW:v5.6.3.024. I have not observed this problem on any other models including the Avaya ERS 5000, 4500, 470 or 460 switches.

Avaya confirmed the presence of the bug today and will be escalating the case to design.

I’m curious if Solarwinds or other management platforms have stumbled upon this bug.

Cheers!

Update: Monday, August 26, 2013

I’ve learned that Avaya will address this bug in software release 5.6.4 which is due out anytime now. ;)

This time around though I quickly found that the sysObjectIDs for the ERS 4800 series switches were missing from SYNTOPICS-ROOT-MIB contained in the most recent software release, v5.6.2.

I received confirmation this morning from Avaya that the SNMP MIBS are missing the proper information and as released don’t include any of the actual sysObjectID OIDs for the Ethernet Routing Switch 4800 switch models.

You can find an updated copy of the SYNOPTICS-ROOT-MIB here. Just replace this file with the SYNOPTICS-ROOT-MIB.mib that is included in the 5.6.2 software release.

You’ll notice that also included are the VSP 7000 and Ethernet Routing Switch 3500 series switches.

This SNMP MIB includes support for the following OIDs;

-- ERS 48xx Series
sreg-ERS-48xx OBJECT IDENTIFIER ::= { registration 78 }
sreg-ERS-4826GTS-PWR-PLUS  OBJECT IDENTIFIER ::= { sreg-ERS-48xx 1 }
sreg-ERS-4850GTS-PWR-PLUS  OBJECT IDENTIFIER ::= { sreg-ERS-48xx 2 }
sreg-ERS-4826GTS           OBJECT IDENTIFIER ::= { sreg-ERS-48xx 3 }
sreg-ERS-4850GTS           OBJECT IDENTIFIER ::= { sreg-ERS-48xx 4 }

-- VSP 7xxx Series
sreg-VSP-7xxx OBJECT IDENTIFIER ::= { registration 79 }
sreg-VSP-7024XLS  OBJECT IDENTIFIER ::= { sreg-VSP-7xxx 1 }

-- ERS 35xx Series
sreg-ERS-35xx OBJECT IDENTIFIER ::= { registration 80 }
sreg-ERS-3526T              OBJECT IDENTIFIER ::= { sreg-ERS-35xx 1 }
sreg-ERS-3526T-PWR-PLUS     OBJECT IDENTIFIER ::= { sreg-ERS-35xx 2 }
sreg-ERS-3524GT             OBJECT IDENTIFIER ::= { sreg-ERS-35xx 3 }
sreg-ERS-3524GT-PWR-PLUS    OBJECT IDENTIFIER ::= { sreg-ERS-35xx 4 }
sreg-ERS-3510GT             OBJECT IDENTIFIER ::= { sreg-ERS-35xx 5 }
sreg-ERS-3510GT-PWR-PLUS    OBJECT IDENTIFIER ::= { sreg-ERS-35xx 6 }

Cheers!

blat.exe -server smtp.acme.org -to Alert@acme.org -cc HelpDesk@acme.org -html -subject "HPOV: UPS on battery ($snn)" -body "<html><body><p style=font-family:Verdana;font-size:12px;font-style:normal;font-weight:normal;><b>HP Open View NNM Alarm Incident Report</b><br /><br />||<b>Date:</b> %date% %time% ($fot)<br />|<b>Alarm:</b> UPS on battery - power failure($name)<br />|<b>Node:</b> $snn($sln)<br />|<b>IP:</b> $mga ($oma)<br />|<b>Contact:</b> $sct<br />|<b>Location:</b> $slc<br /><br />|<b>Notes:</b> generated by HP Open View NNM 9i management server.<br /></p>|</body></html>"

The command line above will utilize blat to send an HTML formatted email message to alert@acme.org with a copy to helpdesk@acme.org with the body looking something similar to the figure to the figure to the right. I’ve sanitized the screenshot to protect the organization I’m currently employed with. The example above is for the SNMP trap ‘upsOnBattery’ while the image to the right is an example of the SNMP trap ‘powerRestored’. You’ll notice the | in the command line is interpreted as a CR/LF by NNM. Here are some of the parameters used in the above example;

• $ssn – node name of the object sending the SNMP trap
• $fot – first occurence time
• $name – OID name of the trap received
• $sln – DNS name of the node
• $mga – management IP address
• $oma – alternative management IP address
• $sct – contact information for the object as stored in sysContact.0
• $slc – location information for the object as stored in sysLocation.0

While it wasn’t too hard it did take some time to get all the formatting down and get it working reliably.

Cheers!

I came across an interesting error today trying to load all the Avaya (formerly Nortel) RAPID-CITY SNMP MIBs. I manually copied all the SNMP MIBs from the 6.2 software release for the Avaya Ethernet Routing Switch 5000 series up to the proper directory, in my case it was D:\ProgramData\HP\HP BTO Software\shared\nnm\user-snmp-mibs\. As a side note “Rapid City” was the development codename for the Ethernet Routing Switch 8600.

I originally tried to load the SNMP MIB from the web GUI but it would continually timeout so I took to the command line interface and issued the following command from a command prompt on the server;

nnmloadmib.ovpl -load "D:\ProgramData\HP\HP BTO Software\shared\nnm\user-snmp-mibs\RC-ERROR-MIB.mib"

I received the following error message from that command;

Error execution program: Error detected while loading MIB File: D:\ProgramData\HP\HP BTO Software\shared\nnm\user-snmp-m
-MIB.mib.
This MIB cannot be loaded until the following problem is corrected:

[RC-ERROR-MIB] - Line 2418: Warning: token longer than allowed maximum, truncating
[RC-ERROR-MIB] - Line 2419: Warning: token longer than allowed maximum, truncating
[RC-ERROR-MIB] - Line 2419: Error defining enumerated type: Missing ')' in 'cannotSetPt2PtOrSpokeTypeWithoutRemoteUniWhi
bled(707'

Accepted syntax for enumerated list:
  { label1(value) [ , label2(value) ... ] }
Examples:
  { other(1), invalid(2), direct(3), indirect(4) }[RC-ERROR-MIB] - Line 2419: Error defining OBJECT-TYPE: last token 'ca
OrSpokeTypeWithoutRemoteUniWhileEndptIsEnabled(707'

Accepted syntax for OBJECT-TYPE:
  object-label OBJECT-TYPE
        SYNTAX type
        [ UNITS "text" ]
        MAX-ACCESS read-only | read-write | read-create | not-accessible
        STATUS current | obsolete | deprecated
  [ DESCRIPTION "text" ]
        [ REFERENCE   "text" ]
  [ INDEX  { variable ...  } ]
  [ DEFVAL { value } ]
  ::= { parentlabel subid }

Example:
  sysUpTime OBJECT-TYPE
        SYNTAX TimeTicks
        MAX-ACCESS read-only
        STATUS current
        DESCRIPTION
          "The time (in hundredths of a second) since the
          network management portion of the system was last
          re-initialized."
  ::= { system 3 }

It turns out that HP Open View NNM doesn’t like tokens that are longer than 60 characters so I had to edit the RC-ERROR-MIB file so that every token was 60 characters or less. If you don’t feel like doing that yourself you can download the file I edited right here, RC-ERROR-MIB

I also had to edit RC-IP-BGP-MIB to fix a different error;

D:\>nnmloadmib.ovpl -load "D:\ProgramData\HP\HP BTO Software\shared\nnm\user-snmp-mibs\RC-IP-BGP.mib"
Error execution program: nnmloadmib: Load file, D:\ProgramData\HP\HP BTO Software\shared\nnm\user-snmp-mibs\RC-IP-BGP.mib,
exist.

D:\>nnmloadmib.ovpl -load "D:\ProgramData\HP\HP BTO Software\shared\nnm\user-snmp-mibs\RC-IP-BGP-MIB.mib"
Error execution program: Error detected while loading MIB File: D:\ProgramData\HP\HP BTO Software\shared\nnm\user-snmp-mib
P-MIB.mib.
This MIB cannot be loaded until the following problem is corrected:

[RC-IP-BGP-MIB] - Line 1529: Error defining OBJECT-TYPE: Expected '::=', found  'The'

Accepted syntax for OBJECT-TYPE:
  object-label OBJECT-TYPE
        SYNTAX type
        [ UNITS "text" ]
        MAX-ACCESS read-only | read-write | read-create | not-accessible
        STATUS current | obsolete | deprecated
  [ DESCRIPTION "text" ]
        [ REFERENCE   "text" ]
  [ INDEX  { variable ...  } ]
  [ DEFVAL { value } ]
  ::= { parentlabel subid }

Example:
  sysUpTime OBJECT-TYPE
        SYNTAX TimeTicks
        MAX-ACCESS read-only
        STATUS current
        DESCRIPTION
          "The time (in hundredths of a second) since the
          network management portion of the system was last
          re-initialized."
  ::= { system 3 }

The fix was to tweak the description used on line 1508, DESCRIPTION “————–“. I replaced it with, DESCRIPTION “Why does this choke within NNM”. If you don’t care to hack the MIB yourself, you can download a fixed copy from here, RC-IP-BGP-MIB

Cheers!

With previous Cisco switches such as the 6509, 3750, 2960, etc we know that the following commands (when sent via a Perl script using the Net-SNMP Perl module) would instruct the switch to copy it’s running-config to a TFTP server.

snmpset -v1 -c$COMMUNITY $HOST ccCopyProtocol.$RANDOM i 1
snmpset -v1 -c$COMMUNITY $HOST ccCopySourceFileType.$RANDOM i 4
snmpset -v1 -c$COMMUNITY $HOST ccCopyDestFileType.$RANDOM i 1
snmpset -v1 -c$COMMUNITY $HOST ccCopyServerAddress.$RANDOM a "10.1.1.50"
snmpset -v1 -c$COMMUNITY $HOST ccCopyFileName.$RANDOM s "sw-train-acme.cfg"
snmpset -v1 -c$COMMUNITY $HOST ccCopyEntryRowStatus.$RANDOM i 1
sleep 5
snmpget -v1 -c$COMMUNITY $HOST ccCopyState.$RANDOM
#if not successful sleep 3 and re-check ccCopyState else continue and destroy table entry
snmpset -v1 -c$COMMUNITY $HOST ccCopyEntryRowStatus.$RANDOM i 6

I know that the both the Cisco Nexus 7010 and 5010 both balk at the SNMP OIDS/MIBS used above. So I’m searching for a set of equivalent SNMP OIDS/MIBS as those in CISCO-CONFIG-COPY-MIB for NX-OS. I’m not sure that such a OID/MIB even exists for NX-OS but it doesn’t hurt to search and ask.

I’m curious if anyone else has come across this issue? I know that there is an XML interface available but I would prefer to keep using the PERL/SNMP script that I’ve already developed. In the interim I’ll probably write an Expect script (or add some Expect code to my existing Perl script) to remotely connect to the switches and issue the appropriate copy commands.

Cheers!

Updated: Monday June 27, 2011

I’ve finally found the issue and now I’m able to backup the Cisco Nexus switches as expected.
[ad name=”ad-articlefooter”]

Here are the values I’m going to be using below;

SNMP v1,v2 read-only string = readme123
SNMP v1,v2 read-write string = writeme123
SNMP v3 userID = Manager (yes I use the same username as the old BayRS software)
SNMP v3 SHA authentication = winnie2009
SNMP v3 AES encryption = poobear2009

Nortel Ethernet Routing Switch 8600

One word of caution with the ERS8600; early default switch configurations included a SNMPv3 user called initial that had full read-write access to the entire switch. I’m not sure if Nortel has changed this behavior but I would strongly urge you to delete any default SNMP v3 users as well as change the default SNMP community strings.

Let’s set the SNMP community strings right away;

 ERS-8610:5# config snmp-v3 community commname first new-commname readme123
 ERS-8610:5# config snmp-v3 community commname second new-commname writeme123

Let’s load the proper AES, 3DES and DES encryption files;

 ERS-8610:5# config load-encryption-module 3DES /flash/p80c5110.img
 ERS-8610:5# config load-encryption-module AES /flash/p80c5110.aes

Let’s create a new SNMP v3 user called Manager;

 config snmp-v3 usm create Manager sha auth winnie priv-prot aes priv poo

Let’s create a new SNMP v3 group called admin;

 config snmp-v3 group-access create admin "" usm authPriv

Let’s give this new group access to the root MIB;

 config snmp-v3 group-access view admin "" usm authPriv read root write root notify root

Let’s add the user Manager to the group admin;

 config snmp-v3 group-member create Manager usm admin

Let’s clear out any previous SNMP trap hosts;

 config snmp-v3 target-addr delete TAddr1
 config snmp-v3 target-addr delete TAddr2

Let’s configure two new SNMP trap hosts. I actually have two configured on all my switches, with one being our HP OpenView Network Node Manager server (10.1.31.1) and the second being our Nortel Enterprise Network Management System server (10.1.31.2);

 config snmp-v3 target-addr create HPOpenView 10.1.31.1:162 TparamV1 taglist trapTag
 config snmp-v3 target-addr create NortelENMS 10.1.31.2:162 TparamV1 taglist trapTag

Let’s delete that default SNMP v3 user just in case it still exists;

 config snmp-v3 usm delete initial

Let’s set the source IP address used to communicate with the SNMP trap hosts. I want this to be the CLIP (Circuitless IP Interface) that I use for all management purposes which in this example is 10.1.50.1. I should mention that the commands below may not appear in switch software earlier than 4.7.1 or 4.6.3.

 config sys set snmp sender-ip 10.1.31.1 10.1.50.1
 config sys set snmp sender-ip 10.1.31.2 10.1.50.1
 config sys set snmp force-trap-sender true
 config sys set snmp force-iphdr-sender true

That should be everything for the Ethernet Routing Switch 8600.

Nortel Ethernet Routing Switch 1600

I’m not going to go into the line by line detail here as I did above. You should be able to follow the explanation provided above.

 ERS-1648T:1# config load-module DES /flash/p16c2160.des

 ERS-1648T:1# config snmp-v3 community commname first new-commname readme123
 ERS-1648T:1# config snmp-v3 community commname second new-commname writeme123

 ERS-1648T:1# config snmp-v3 usm create Manager sha auth winnie2009 priv poobear2009
 ERS-1648T:1# config snmp-v3 group-access create admin "" usm authPriv
 ERS-1648T:1# config snmp-v3 group-access view admin "" usm authPriv read root write root notify root
 ERS-1648T:1# config snmp-v3 group-member create Manager usm admin
 ERS-1648T:1# config snmp-v3 target-addr create HPOpenView 10.1.31.1:162 TparamV1 taglist trapTag
 ERS-1648T:1# config snmp-v3 target-addr create NortelNMS 10.1.31.2:162 TparamV1 taglist trapTag

 ERS-1648T:1# config snmp-v3 usm delete initial

That’s the ERS1600 series switch.

Nortel Ethernet Routing Switch 4500, 5500, 5600 Series

We need to create a new view so we’ll use the name snmpView;

 5520-48T-PWR(config)# snmp-server view snmpView +1.3

If you have the secure image loaded then you have access to SHA authentication, DES, 3DES and AES encryption.

 5520-48T-PWR(config)# snmp-server user Manager sha winnie2009 aes poobear2009 read-view snmpView write-view snmpView notify-view snmpView

If you receive an error using the command above (see below) you may not have the secure software image loaded on the switch. If you want to use SHA authentication, DES, 3DES or AES encryption you’ll need to load the secure image. Example SW:v6.1.0.006 will only allow you to use the md5 authentication with no encryption while SW:v6.1.0.007 will allow both MD5 and SHA authentication along with DES, 3DES or AES encryption.

 5520-48T-PWR(config)#snmp-server user Manager sha winnie2009 aes poobear2009 read-view snmpView write-view snmpView notify-view snmpView
snmp-server user Manager sha winnie2009 aes poobear2009 read-view snmpView writ
                         ^
% Invalid input detected at '^' marker.

You can use the following command to just use MD5 authentication with no encyrption;

 5520-48T-PWR(config)#snmp-server user Manager md5 winnie2009 read-view snmpView write-view snmpView notify-view snmpView

Java Device Manager

With all that done you can now use Nortel’s Java Device Manager to manage the switch using SNMP v3.

In the example to the left I’m going to connect to the ERS8600 switch at the management IP address of 10.1.50.1.

We are going to use the SNMP v3 user of Manager which we configured above.

We will also use the Authentication Protocol of SHA-96 using the Authentication Password of winnie2009 which we configured above.

We will use the Privacy Protocol of AES which we configured above along with the Privacy Password of poobear2009.

Cheers
[ad name=”ad-articlefooter”]

Today I’ve a little challenge on my network: configure a permission to a specific IP for read the temperature of two ERS8600. This specific host don’t become part of my management network, so I can’t use the same snmp read community. I don’t like to free everything on the core to be read, so I start to liberate only the specific OID (temperature of chassis) on my two ERS8600, and only for the specific IP of the host, with a new read community.

After some study on Nortel documentation (2008_04_04_SNMP_on_ERS_8600_TCG_NN48500564.pdf) I present us my little todo for everone that needs some similar, because this document is not the mos objective guide of the world. My steps:

Step1: Create a MIB view, called “only_temp”, restricted for the temperature OID:

config snmp-v3 mib-view create only_temp 1.3.6.1.4.1.2272.1.100.1.2.0 type include

View the changes:

config snmp-v3 mib-view info

Step2: Create a access group called “group_temp”, with snmpv1 and v2c, no authentication, reading the “only_temp” mib-view:

config snmp-v3 group-access create group_temp "" snmpv1 noAuthNoPriv
config snmp-v3 group-access create group_temp "" snmpv2c noAuthNoPriv
config snmp-v3 group-access view group_temp "" snmpv1 noAuthNoPriv read only_temp write only_temp
config snmp-v3 group-access view group_temp "" snmpv2c noAuthNoPriv read only_temp write only_temp

View the changes:

config snmp-v3 group-access info

Step3: Create the user “user_temp” inside the group:

config snmp-v3 group-member create user_temp snmpv1 group_temp
config snmp-v3 group-member create user_temp snmpv2c group_temp

View the changes:

config snmp-v3 group-member info

Step4: Create a new community “ers8600”, index “third” (the first and second already exist, adapt for you scenario), for the user “user_temp”

config snmp-v3 community create third ers8600 user_temp

View the changes:

config snmp-v3 community info

Step5: Create a new access-policy (policy 6 in my case) for the specific IP 10.10.10.1 (where the temperature has been monitored):

config sys access-policy policy 6 create
config sys access-policy policy 6 name policy6
config sys access-policy policy 6 accesslevel ro
config sys access-policy policy 6 network 10.10.10.1/255.255.255.255
config sys access-policy policy 6 snmp-group-add group_temp snmpv1
config sys access-policy policy 6 snmp-group-add group_temp snmpv2c
config sys access-policy policy 6 service telnet disable
config sys access-policy policy 6 service ssh disable
config sys access-policy policy 6 service tftp disable
config sys access-policy policy 6 service ftp disable
config sys access-policy policy 6 service snmpv3 enable

I hope this can help someone. Bye!

I think this was a great post and appreciate Forrequi sharing this with everyone!

Cheers!

You can certainly do this from Nortel’s Java Device Manager, however, you need to be careful that you don’t saw off the branch you’re standing on when you change the SNMP community string. It’s best to configure the SNMP community strings from the CLI interface to avoid any potential issues.

Here are the CLI commands to configure the SNMP community strings on the ERS 8600 and 1600 switch. In the example below we’ll set the read-only string to open and the read-write string to lock.

ERS-8610:5# config snmp-v3 community commname first new-commname open
ERS-8610:5# config snmp-v3 community commname second new-commname lock

Here are the CLI commands to configure the SNMP community strings on the ERS 4500, ERS 5500 and ES460/470 switches. In the example below we’ll set the read-only string to open and the read-write string to lock.

5520-48T-PWR (config)# snmp-server community open ro
5520-48T-PWR (config)# snmp-server community lock rw

Cheers!

The script has been tested and works on Nortel’s BayRS routers (ARN, ASN, BLN, BCN). You just obviously need to be careful of how the script interprets the ipNetToMediaIfIndex value depending on the device you are polling.

The script get8600arp.pl is a very straight forward script. It simply polls various SNMP OIDs and then stores the results in a file. It does this for every switch (FQDN/IP Address) that is listed in the input file.

#!/usr/bin/perl
#
# Filename: /root/get8600arp.pl
#
# Purpose:  Query Nortel Ethernet Routing Switch 8600 for the IP ARP
#           table via SNMP. This script will poll a list of devices
#           (input file) and dump the contents of the IP ARP table to
#           and outputfile.
#
# Author:   Michael McNamara
#
# Date:     December 5, 2002
#
# Support Switches:
#           - Nortel ERS 8600
#           - Nortel ERS 1600
#           - Nortel ERS 5500
#           - Nortel BayRS Routers
#
# Requirements:
#           - Net-SNMP
#           - Net-SNMP Perl Module
#           - SNMP-MIBS
#
# Changes:
#
#           - May  5, 2007 (M.McNamara)
#           clean up code and documentation for release to public
#           - Oct 10, 2006 (M.McNamara)
#           went back to SNMP v1 to support BayRS legacy routers
#           - Sep 04, 2003 (M.McNamara)
#           migrated from vendor specific MIB to RFC1213 (ipNetToMediaNetAddress)
#

# Load Modules
use strict;
use SNMP;
use Net::Ping;

# Declare constants
#use constant DEBUG      => 0;           # DEBUG settings
use constant RETRIES    => 3;           # SNMP retries
use constant TIMEOUT    => 1000000;     # SNMP timeout, in microseconds
use constant SNMPVER    => 1;           # SNMP version

# SNMP Settings
$SNMP::verbose = 0;
$SNMP::use_enums = 1;
$SNMP::use_sprint_value = 0;
&SNMP::initMib();
&SNMP::loadModules('RAPID-CITY');

# Declaration Variables
my ($sess, @vals);
my @devices;
my ($card, $port);
my $snmphost;
my $comm = "public";        # SNMP ReadOnly Community String
my %array;
my $switchfile;
my $datafile;

our $DEBUG;                     # DEBUG flag

undef @devices;

# Program and help information
my $program = "get8600arp.pl";
my $version = "v1.3";
my $author = "Michael McNamara";
my $purpose = "This Perl script is retreieve the IP ARP table from the ERS8600 Layer 3 switch/router and store it in file for later use.";
my $usage = "Usage: $program \[input\] \[output\] \[-help\] \[debug\]\n    <input>  = filename listing each switch to poll\n    <output> = filename where to store output\n";

if (($#ARGV +1) <= 2) {
 print "Program: $program \nVersion: $version \nWritten by: $author \n$purpose\n\n$usage\n";
 print "DEBUG: ARGV =  $#ARGV\n";
 print "DEBUG: ARGV =  $ARGV[0] $ARGV[1] $ARGV[2] $ARGV[3]\n";
 exit;
}

my $arg1 = shift @ARGV;
my $arg2 = shift @ARGV;
my $arg3 = shift @ARGV;

if ($arg1 =~ /help/) {
 print "Program: $program \nVersion: $version \nWritten by: $author \n$purpose\n\n$usage\n";
 print "DEBUG: ARGV =  @ARGV\n";
 print "DEBUG: ARGV =  $ARGV[0] $ARGV[1] $ARGV[2] $ARGV[3]\n";
 exit;
}

$switchfile = $arg1;
$datafile = $arg2;
$DEBUG = $arg3;

# Test to see if inputifle exists
if (!-e $switchfile) {
 die "ERROR: Unable to locate and/or open inputfile $switchfile...";
}

############################################################################
##### B E G I N   M A I N ##################################################
############################################################################

&load_switches;

&collect_arp;

exit 0;

############################################################################
#### E N D   M A I N #######################################################
############################################################################

############################################################################
# Subroutine collect_arp
#
# Purpose: collect ARP information from layer 3 switches/routers
############################################################################
sub collect_arp {

 # Open output datafile for appending
 open(DATAFILE, ">>$datafile");

 # Loop over each Passport 8600 switch
 foreach $snmphost (@devices) {

    my $packet = Net::Ping->new('icmp');

    $snmphost =~ s/\n//g;        # remove CRLF

    if ($packet->ping($snmphost)) {

       $sess = new SNMP::Session (    DestHost   =>  $snmphost,
                              Community  =>  $comm,
                              Retry      =>  RETRIES,
                              Timeout    =>  TIMEOUT,
                              Version    =>  SNMPVER );

       my $vars = new SNMP::VarList(
                              ['ipNetToMediaIfIndex', 0],
                              ['ipNetToMediaPhysAddress', 0],
                              ['ipNetToMediaNetAddress', 0],
                              ['ipNetToMediaType', 0] );

       while (1) {

          @vals = $sess->getnext($vars);  # retreive SNMP information

          last unless ($vars->[0]->tag eq 'ipNetToMediaIfIndex');

          $vals[1] = unpack('H12', $vals[1]);
          $vals[1] =~ tr/a-z/A-Z/;

          $card = (($vals[0] & 62914560) / 4194304);
          $port = (($vals[0] & 4128768) / 65536) + 1;

          print "$snmphost, $vals[0], ($card/$port), $vals[1], $vals[2], $vals[3]\n" if ($DEBUG);
          print DATAFILE "$snmphost, $vals[0], $card, $port, $vals[1], $vals[2]\n";

          $array{$snmphost}[$card][$port] = $vals[2];

       } # end while

    } else {

       print ("ERROR: $snmphost not responding to ICMP ping skipping...\n");

    } #end if $packet

 } #end foreach

 close(DATAFILE);

} #end sub collect_arp

############################################################################
# Subroutine load_switches
#
# Purpose: load list of switches
############################################################################
sub load_switches {

 open(SWITCHLIST, "<$switchfile");

 # Walk through data file
 while (<SWITCHLIST>) {

    # Skip blank lines
    next if (/^\n$/);
    # Skip comments
    next if (/^#/);

    #print "DEBUG: adding $_ to our list of devices \n" if ($DEBUG);

    push (@devices, $_);

 }

 close(SWITCHLIST);

 return 1;

} # end sub load_switches
############################################################################

The real magic that folks have always been searching for is the binary formula to turn the ipNetToMediaIfIndex into a location that denotes the card and port where that specific device is connected to.

$card = (($vals[0] & 62914560) / 4194304);
$port = (($vals[0] & 4128768) / 65536) + 1;

While I still use flat files you could certainly adopt this code to dump the output into a database. I just haven’t had the time although I’ve been playing with MySQL quite a bit lately.

Cheers!

In the post I was responding to someone looking for information on how to decode the value returned from the ipNetToMediaIfIndex when querying an ERS 8600 switch. Thankfully Shane (Nortel) was able to help me come up with the forumla.

card = ( $value AND 62914560 ) / 4194304
port = (( $value AND 4128768) / 65536 ) + 1

With that formula you could now walk the ipNetToMediaTable and retreieve the entire ARP table providing you the card and port number, MAC address, and IP address for each entry in the table.

The next issue was how to deal with MultiLink Trunk interfaces. In this case (and with my current software code) I build a table of all the MLT interfaces prior to polling the ipNetToMediaTable. I still use Perl but it shouldn’t be very hard to convert to PHP.

# rcMltNumMlts
$nummlts = $sess->get("rcMltNumMlts.0");

for ($i = 1; $i <= $nummlts; $i++) {
  # rcMltName         
  $mltname[$i] = $sess->get("rcMltName.$i");
  # rcMltId
  $mltindex[$i] = $sess->get("rcMltId.$i");
  # rcMltIfIndex
  $mltifindex[$i] = $sess->get("rcMltIfIndex.$i");
  print "DEBUG: MltId = $i and MltName = $mltname[$i] and MltIndex = $mltindex[$i] and MltIfIndex = $mltifindex[$i]\n" if ($DEBUG);
};

Now that we have the rcMltTable in an array we can walk the ipNetToMediaTable and match up any entries. Here’s the code I use (again it’s Perl but you should be able to convert to PHP);

# Evaulate with bitwise operation
$card = (($vals[0] & 62914560) / 4194304);
$port = (($vals[0] & 4128768) / 65536) + 1;

# Evaulate to determine if port is a MLT
if ($card != 0) {
  $intf = (((64 * $card) + $port) - 1);
  print "DEBUG: $vals[1] address found on card $card port $port\n";
} else {
  $mlt = 1;
  print "DEBUG: $vals[1] address found on MLT $mltname[$port]\n";
} # end else

Hopefully that doesn’t look too complicated. The important piece here is that you need to merge the rcMltTable with the ipNetToMediaTable to get your results. If you name the MLT with something meaningful you can then return that string to the application that is making the query.

I wrote a Perl application that would search the ARP table of an Ethernet Routing Switch 8600 dynamically for a specific IP address entry. Here’s an example of the output;

Nortel Passport 8600 Gigabit Switch IP ARP Table Search

Initializing query for sw-ccr-8600.datacenter.acme.org for IP address 1.1.1.10...

sysDescr = ERS-8610 (4.1.3.0)
sysObjectID = .1.3.6.1.4.1.2272.30
sysUpTime = 169 Days 6 Hours 43 mins 11 secs
sysContact = Acme Network Infrastructure Team
sysName = sw-ccr-8600.datacenter.acme.org
sysLocation = USA

Please be patient it may take a while to complete the search...

DEVICE FOUND

1.1.1.10 (000AE4753FC9) address found on MLT SMLT-5500

We searched through 1183 forwarding records...

That's all folks!

I will look to publish the complete code on my website sometime in the near future.

Cheers!

I’ve decided to post some of the vendor specific SNMP MIBS that I work with on my homepage. You should be able to link straight to my homepage with this URL;

http://blog.michaelfmcnamara.com/mibs/

You should be able to find SNMP MIBS for the following devices;

Nortel Ethernet Routing Switch 8600 (v4.1.4)
Nortel Ethernet Routing Switch 5500 Series (v5.1)
Motorola WS5100 Wireless LAN Switch (v3.0.3)
Motorola RFS7000 Wireless LAN Switch (v1.x)
APC UPS Management Cards (v387)

As time and disk space allow I will add additional vendor MIBS and additional devices.

Update 12/01/07

Polycom VXS8000 Video Conferencing System
Blue Coat ProxySG Appliance
Blue Coat ProxyAV Appliance

Update 12/07/07

Nortel Application Switch (v23.2.3.1)

Update 12/26/07

Nortel Ethernet Switch 460/470 (v3.7)
Nortel Ethernet Routing Switch 1600 (v2.1.4)
Nortel Succession Call Server (v4.5)

Update 12/29/2007

Motorola WS5000/WS5100 Wireless LAN Switch (v2.1.3)

Cheers!

Update July 24, 2008

I’ve moved the files to my new host and changed the pointers on this page.

Cheers!

Hopefully everyone out there is backing up their network switch configurations in the unlikely event that if their hardware dies they only need to worry about replacing the hardware and not about re-configuring the entire switch.

Quite a few years back I wrote a Perl script that would send the proper SNMP commands to instruct a network switch to copy it’s configuration to a TFTP server. This script essentially became known as “switchtftpbackup.pl” It’s nothing fancy or pretty but it gets the job done.

I’ve posted this Perl script on my webiste under the Perl section.

I run this script from Cron one of our CentOS Linux servers at work every week. The same server also acts as a central TFTP server for the entire organization. I also run other scripts that then archive the weekly backups, in the event that I need to go to a backup that’s more than a week old.

I believe both Nortel’s Optivity NMS and Cisco’s Cisco Works both have options to backup switch configurations these days.

What are you using?

Cheers!