Michael McNamara

technology, networking, virtualization and IP telephony

ADAC and VLAN Configurations

by

We’ve just recently come across this problem and I thought it would be a great topic to share and perhaps even solicit some feedback from others. As you might already know I’ve been deploying ADAC across a large number of Nortel Ethernet Routing Switch 5520s with great success. ADAC allows the switch to control the phones voice VLAN configuration.

Well we also ran into a problem after upgrading a number of those switches to v5.1.1.17. A network administrator had made VLAN changes to various ports on the switch prior to the upgrade but after ADAC had been enabled on the ports. After the upgrade the switch ports defaulted back to the original VLAN they were configured for when ADAC was first enabled. We performed some additional testing and found that this problem would occur if the switch was just reset (rebooted) so it doesn’t appear to be tied to the upgrade but rather the action of restarting the switch. Looking at how ADAC works I can understand the problem but I’m disappointed that Device Manager or the CLI interface doesn’t throw a warning when you try to change the VLAN configuration of a port with ADAC enabled.

The lesson here is that you should disable ADAC on any port where you intend to change the VLAN membership.

Anyone else seen this?

Cheers!

Link Flap and Succession Voice Gateway Media Cards

by

The Nortel Ethernet Routing Switch 8600 supports a feature called link flap detect. This feature is designed to detect a port whose link is flapping up and down and potentially disable the port and/or issue an SNMP trap. The feature allows the configuration of a specific frequency and interval which can be used to fine tune control of the feature.

A few years ago we deployed about 25 Voice Gateway Media Cards (VGMC) across five different Nortel Call Server 1000Ms. We almost immediately started to notice a problem between the VGMCs and the ERS 8600 switches. The VGMCs would link up and link down multiple times as they booted up. While there is technically no real problem with the VGMC you can run straight into a road block if you reboot the card multiple times in a short period of time. Without getting to crazy on the explanation the ERS 8600 switch would potentially disable the switch port(s) that connected to the VGMCs if they were rebooted to often during a 60 minute period. The problem was only observed while the VGMCs were booting up, once the VGMC was up and running the problem never occurred. I believe the link flap detect feature is enabled by default unless specifically disabled.

This may be an old problem that might have already been corrected by Nortel but I still have the link flap detect feature disabled on the Ethernet Routing Switch 8600s that connect to our Nortel Voice Gateway Media Cards.

ERS8600:5# config sys link-flap-detect interval 60
ERS8600:5# config sys link-flap-detect frequency 10
ERS8600:5# config sys link-flap-detect auto-port-down disable
ERS8600:5# config sys link-flap-detect send-trap enable

The configuration can be confirmed using the following command;

ERS8600:5# show sys link-flap-detect general-info

Auto Port Down: enable
Send Trap : enable
Interval : 60
Frequency : 10

The frequency defines how often a port can go down, where the default is 10 (times). The default interval is 60 minutes; therefore the port will be disabled if the port goes down 10 times within 60 minutes.

Cheers!

ERS 8600 Boot Configuration Sequence

by

The Nortel Ethernet Routing Switch 8600 will automatically search for several boot configuration files as it boots up. It will look for several files in a specific order;

  1. /pcmcia/pcmboot.cfg
  2. /flash/boot.cfg
  3. /pcmcia/boot.cfg

The file /pcmcia/pcmboot.cfg is designed to allow network engineers to deploy the ERS 8600 remotely instructing local site personnel to simply copy a file (pcmboot.cfg) that could be emailed to them to the PCMCIA flash card, inserted it into the switch and the switch booted up. I’ve never personally used this feature but I have remotely installed several ERS 8600s by having a modem attached to the console port.

What would a normal boot.cfg file look like?

Here’s the current standard that I’m using on the 37 ERS 8600 switches that I maintain.

#
# TUE JUL 22 07:25:38 2008 EDT
# box type : 8k boot configuration file
#
flags daylight-saving-time true
flags ftpd true
flags savetostandby true
flags telnetd true
flags tftpd true
flags verify-config false
tz dst-end M11.1.0/0200
tz dst-name "EDT"
tz dst-start M3.2.0/0200
tz name "EST"
tz offset-from-utc 300
choice primary image-file "/flash/p80a4163.img"

The commands should be self explanatory for the most part. The last line of the file instructs the switch which agent code it should boot, in this example I’m running 4.1.6.3 software so we’ll boot /flash/p80a4163.img.

Cheers!

VoIP USB Adapter MagicJack

by

A few weeks ago a few friends approached me about Internet based VoIP solutions for their home phone. They were fed up with the $100.00+ phone bills and weren’t really excited about giving any more money to the local cable television company(Comcast). I’ve been an AT&T CallVantage VoIP customer for the past 2-3 years so I was obviously ready to recommend AT&T CallVantage until I discovered that they are no longer accepting new customers. While I wasn’t ready to recommend Vonage or any of the other solutions out there I did comment to them about the recent buzz around a product called MagicJack. The MagicJack USB adapter itself costs approximately $39.95 and includes the first year of service free while subsequent years are $19.95 a year (yes you read that right $19.95/year). The solution requires a Windows XP or MacOS desktop/laptop and utilizes your broadband Internet connection. I personally know of two folks that are currently utilizing the solution and they absolutely love it and they are admittingly not very technical or computer savy. However, they simply love the solution and they both estimate that it’s saving them between $75 and $100 a month in long distance phone charges. The solution has scored numerous product awards including PC Magazine’s Editor’s Choice award.

So while I’m not exactly sure what I’ll do myself since it’s probably only a matter of time until AT&T pulls the plug on CallVanage it seems like MagicJack could be a great solution for those teenagers heading off to college. They’d no longer have an excuse for not calling home every once-n-while. :)

Cheers!

Ethernet Routing Switch 8600 Log files

by

The Nortel Ethernet Routing Switch 8600 has multiple log files that can be examined for signs of a problem or during basic troubleshooting. The log files will be stored on the PCMCIA flash card or in memory if there is no flash card available. There is also a trace facility within the ERS 8600 that allows you to log extensive debug information within the switch on different subsystems and processes. I won’t go into the trace facility in this post but if your interested you can find it in by using the command “config trace” from the CLI interface of the ERS 8600 switch.

You can check the contents of the flash memory and PCMCIA cards by issuing the “dir” command;

ERS-8610:5# dir
  size          date       time       name
--------       ------     ------    --------
 8773031    JUL-22-2008  06:11:38   /flash/p80a4163.img
     373    JUL-22-2008  06:30:36   /flash/boot.cfg
       8    AUG-16-2008  13:28:10   /flash/ospf_md5key.txt
      79    AUG-16-2008  13:28:10   /flash/snmp_comm.txt
   16440    AUG-16-2008  13:28:14   /flash/config.cfg
      11    JUL-22-2008  07:33:30   /flash/engboot
 1073634    JUL-22-2008  06:11:44   /flash/p80b4163.img
   55928    JUL-22-2008  06:11:44   /flash/p80c4163.img
   26112    JUL-22-2008  06:11:46   /flash/p80c4163.aes
 8872259    JUL-22-2008  06:16:20   /flash/p80m4163.img
 1272852    JUL-22-2008  06:17:12   /flash/p80j4163.dld
total: 64155648 used: 20639744 free: 43515904 bytes
  size          date       time       name
--------       ------     ------    --------
   29671    AUG-16-2008  14:03:18   /pcmcia/7b700005.000
   26192    AUG-16-2008  14:16:48   /pcmcia/clilog.txt
       1    JUL-22-2008  06:21:20   /pcmcia/clilogoff
       8    JUL-22-2008  07:33:30   /pcmcia/7b700005.num
total: 64710656 used: 584704 free: 64125952 bytes

You can examine the log files with the following commands;

show log file tail

ERS-8610:5# show log file tail
CPU5 [07/22/08 07:44:05] SNMP INFO Communication established with backup CPU
CPU5 [07/22/08 07:44:05] HW INFO System activity performed
CPU5 [07/22/08 07:43:44] SNMP INFO CPU switch over, stand-by CPU in slot # 5 became master
CPU5 [07/22/08 07:43:44] SNMP INFO Sending Warm-Start Trap

CPU5 [07/22/08 07:43:44] SNMP INFO CPU switch over, stand-by CPU becoming master
CPU5 [07/22/08 07:43:37] OSPF INFO Ospf Nbr State Change: rtid:10.7.1.9, ipa:10.7.1.5 nbr-rtid:10.7.0.1  LOADING->FULL Event LOADING_DONE_EVENT
CPU5 [07/22/08 07:43:37] OSPF INFO Ospf Nbr State Change: rtid:10.7.1.9, ipa:10.7.1.6 nbr-rtid:10.7.0.2  LOADING->FULL Event LOADING_DONE_EVENT
CPU5 [07/22/08 07:43:32] OSPF INFO Ospf Nbr State Change : rtid:10.7.1.9, ipa:10.7.1.6 nbr-rtid:10.7.0.2 EXCHANGE->LOADING Event EXCHANGE_DONE_EVENT
CPU5 [07/22/08 07:43:32] OSPF INFO Ospf Nbr State Change : rtid:10.7.1.9, ipa:10.7.1.5 nbr-rtid:10.7.0.1 EXCHANGE->LOADING Event EXCHANGE_DONE_EVENT
CPU5 [07/22/08 07:43:32] OSPF INFO Ospf Nbr State Change : rtid:10.7.1.9, ipa:10.7.1.5 nbr-rtid:10.7.0.1 EX_START->EXCHANGE Event NEGOTIATION_DONE_EVENT
CPU5 [07/22/08 07:43:32] OSPF INFO Ospf Nbr State Change : rtid:10.7.1.9, ipa:10.7.1.6 nbr-rtid:10.7.0.2 EX_START->EXCHANGE Event NEGOTIATION_DONE_EVENT
CPU5 [07/22/08 07:43:32] OSPF INFO Ospf Nbr State Change : rtid:10.7.1.9, ipa:10.7.1.5 nbr-rtid:10.7.0.1 2WAY->EX_START Event ADJ_OK_EVENT
CPU5 [07/22/08 07:43:32] OSPF INFO Ospf Nbr State Change : rtid:10.7.1.9, ipa:10.7.1.6 nbr-rtid:10.7.0.2 2WAY->EX_START Event ADJ_OK_EVENT
CPU5 [07/22/08 07:43:32] OSPF INFO Ospf Nbr State Change : rtid:10.7.1.9, ipa:10.7.1.6 nbr-rtid:10.7.0.2 INIT->2WAY Event TWO_WAY_EVENT
CPU5 [07/22/08 07:43:32] OSPF INFO Ospf Nbr State Change : rtid:10.7.1.9, ipa:10.7.1.6 nbr-rtid:10.7.0.2 DOWN->INIT Event HELLO_RECEIVED_EVENT
CPU5 [07/22/08 07:43:32] OSPF INFO Ospf Nbr State Change : rtid:10.7.1.9, ipa:0.0.0.0 nbr-rtid:10.7.0.2 NULL->DOWN Event HELLO_RECEIVED_EVENT
CPU5 [07/22/08 07:43:32] OSPF INFO Ospf Nbr State Change : rtid:10.7.1.9, ipa:10.7.1.5 nbr-rtid:10.7.0.1 INIT->2WAY Event TWO_WAY_EVENT
CPU5 [07/22/08 07:43:32] OSPF INFO Ospf Nbr State Change : rtid:10.7.1.9, ipa:10.7.1.5 nbr-rtid:10.7.0.1 DOWN->INIT Event HELLO_RECEIVED_EVENT
CPU5 [07/22/08 07:43:32] OSPF INFO Ospf Nbr State Change : rtid:10.7.1.9, ipa:0.0.0.0 nbr-rtid:10.7.0.1 NULL->DOWN Event HELLO_RECEIVED_EVENT
CPU5 [07/22/08 07:43:32] SNMP INFO Spanning Tree Topology Change(StgId=1, PortNum=2/1, MacAddr=00:1d:42:7b:70:01)
CPU5 [07/22/08 07:43:02] NONE INFO Spanning Tree Topology Change. New Root bridge 00:04:38:70:70:01 for StgId = 1
CPU5 [07/22/08 07:43:02] SNMP INFO Link Up(2/1)
CPU5 [07/22/08 07:43:00] SNMP INFO Vlacp link up(1/1)
CPU5 [07/22/08 07:43:00] SNMP INFO Vlacp link down(1/1)
CPU5 [07/22/08 07:43:00] SNMP INFO Link Up(1/2)
CPU5 [07/22/08 07:43:00] SNMP INFO Link Up(1/1)
CPU5 [07/22/08 07:43:00] SW INFO PCMCIA card detected in Master CPU "ERS-8610" slot 5, Chassis S/N SSPNDTxxxx
CPU5 [07/22/08 07:43:00] SNMP INFO Fan Up(FanId=2, OperStatus=2)
CPU5 [07/22/08 07:43:00] SNMP INFO Fan Up(FanId=1, OperStatus=2)
CPU5 [07/22/08 07:43:00] HW INFO System activity performed
CPU5 [07/22/08 07:43:01] SNMP INFO Vlacp link down(1/1)
CPU5 [07/22/08 07:43:01] SNMP INFO Booted with PRIMARY boot image source - /flash/p80a4163.img
CPU5 [07/22/08 07:43:01] SW INFO The system is ready
CPU5 [07/22/08 07:43:01] SW INFO NTP Enabled
CPU5 [07/22/08 07:43:00] SNMP INFO 2k card up(CardNum=2 AdminStatus=1 OperStatus=1)
CPU5 [07/22/08 07:43:00] SW INFO Loading configuration from /flash/config.cfg
CPU5 [07/22/08 07:43:00] HW INFO FFAD:Test Passed OK Slot:6 Tap:1
[07/22/08 07:43:00] The previous message repeated 1 time(s).
CPU5 [07/22/08 07:43:00] HW INFO FFAD Setup: Serializer-Deserializer Connection OK Slot:6 Tap:1
CPU5 [07/22/08 07:43:00] HW INFO FFAD:Test Passed OK Slot:5 Tap:1
[07/22/08 07:43:00] The previous message repeated 1 time(s).
CPU5 [07/22/08 07:43:00] HW INFO FFAD Setup: Serializer-Deserializer Connection OK Slot:5 Tap:1
CPU5 [07/22/08 07:43:00] SNMP INFO 2k card up(CardNum=1 AdminStatus=1 OperStatus=1)
CPU5 [07/22/08 07:43:00] HW INFO Initializing 8648GTR in slot #2 ...
CPU5 [07/22/08 07:42:59] HW INFO FFAD:Test Passed OK Slot:6 Tap:0
[07/22/08 07:42:59] The previous message repeated 1 time(s).
CPU5 [07/22/08 07:42:59] HW INFO FFAD Setup: Serializer-Deserializer Connection OK Slot:6 Tap:0
CPU5 [07/22/08 07:42:59] HW INFO FFAD:Test Passed OK Slot:5 Tap:0
[07/22/08 07:42:59] The previous message repeated 1 time(s).
CPU5 [07/22/08 07:42:59] HW INFO FFAD Setup: Serializer-Deserializer Connection OK Slot:5 Tap:0
CPU5 [07/22/08 07:42:59] HW INFO Initializing 8630GBR in slot #1 ...
CPU5 [07/22/08 07:42:59] SW INFO slot 1 found NP heartbeat - R-Module is online
CPU5 [07/22/08 07:42:57] SW INFO slot 2 found NP heartbeat - R-Module is online
CPU5 [07/22/08 07:42:52] SW INFO Slot  2: Starting software version 4.1.6.3
CPU5 [07/22/08 07:42:51] SW INFO Slot  1: Starting software version 4.1.6.3
CPU5 [07/22/08 07:42:41] SW INFO Slot  2: Loading /flash/p80j4163.dld
CPU5 [07/22/08 07:42:41] SW INFO Slot  1: Loading /flash/p80j4163.dld
CPU5 [07/22/08 07:42:36] SNMP INFO 2k card up(CardNum=6 AdminStatus=1 OperStatus=1)
CPU5 [07/22/08 07:42:36] HW INFO FFAD:Test Passed OK Slot:6 Tap:8
CPU5 [07/22/08 07:42:36] HW INFO FFAD Setup: Serializer-Deserializer Connection OK Slot:6 Tap:8
CPU5 [07/22/08 07:42:34] SNMP INFO 2k card up(CardNum=5 AdminStatus=1 OperStatus=1)
CPU5 [07/22/08 07:42:34] HW INFO Initializing 8692SF in slot #6 ...
CPU5 [07/22/08 07:42:34] HW INFO FFAD:Test Passed OK Slot:5 Tap:8
CPU5 [07/22/08 07:42:34] HW INFO FFAD Setup: Serializer-Deserializer Connection OK Slot:5 Tap:8
CPU5 [07/22/08 07:42:32] HW INFO Initializing 8692SF in slot #5 ...
CPU5 [07/22/08 07:42:32] HW INFO Card inserted: Slot=6 Type=8692SF
CPU5 [07/22/08 07:42:32] SW INFO R-Module inserted: Slot=2 Type=8648GTR, waiting to bootup...
CPU5 [07/22/08 07:42:32] SW INFO R-Module inserted: Slot=1 Type=8630GBR, waiting to bootup...
CPU5 [07/22/08 07:42:32] HW INFO Card inserted: Slot=5 Type=8692SF
CPU5 [07/22/08 07:42:30] SW INFO Killing task tCXCTask after timeout
CPU5 [07/22/08 07:42:28] SW INFO Killing task tChRxTask after timeout
CPU5 [07/22/08 07:42:26] HW INFO Stand-by CPU in slot # 5 becoming master...
CPU5 [07/22/08 07:33:34] HW INFO System activity performed
CPU5 [07/22/08 07:33:32] SW INFO PCMCIA card detected in Stand-by CPU "ERS-8610" slot 5, Chassis S/N SSPNDTxxxx
CPU5 [07/22/08 07:33:32] SW INFO Loading configuration from /flash/config.cfg
CPU5 [07/22/08 07:33:30] SW INFO CPU card entering warm-standby mode...
CPU5 [07/22/08 07:33:30] SW INFO ERS System Software Release 4.1.6.3
CPU5 [07/22/08 07:33:30] SW INFO System boot

If you have CLI logging enabled you can dump that log with the following command;

show clilog file tail

ERS-8610:5# show clilog file tail
Slot5     5 [07/28/08 21:47:29] TELNET:10.1.198.53 rwa show log file tail
Slot5     4 [07/28/08 21:47:27] TELNET:10.1.198.53 rwa box
Slot5     3 [07/28/08 21:47:25] TELNET:10.1.198.53 rwa show log file tail
Slot5     2 [07/28/08 21:47:09] TELNET:10.1.198.53 rwa info
Slot5     1 [07/28/08 21:47:08] TELNET:10.1.198.53 rwa show ip route
Slot5    62 [07/22/08 07:30:47] TELNET:10.1.20.1 rwa boot /flash/p80b4163.img
Slot5    61 [07/22/08 07:30:23] TELNET:10.1.20.1 rwa dir
Slot5    60 [07/22/08 07:30:15] TELNET:10.1.20.1 rwa peer telnet
Slot5    59 [07/22/08 07:26:36] TELNET:10.1.20.1 rwa peer telnet
Slot5    58 [07/22/08 07:26:34] TELNET:10.1.20.1 rwa box
Slot5    57 [07/22/08 07:26:30] TELNET:10.1.20.1 rwa info
Slot5    56 [07/22/08 07:26:28] TELNET:10.1.20.1 rwa ?
Slot5    55 [07/22/08 07:26:05] TELNET:10.1.20.1 rwa more /flash/config.cfg
Slot5    54 [07/22/08 07:25:54] TELNET:10.1.20.1 rwa info
Slot5    53 [07/22/08 07:25:50] TELNET:10.1.20.1 rwa more /flash/boot.cfg
Slot5    52 [07/22/08 07:25:41] TELNET:10.1.20.1 rwa save config
Slot5    51 [07/22/08 07:25:38] TELNET:10.1.20.1 rwa save bootconfig
Slot5    50 [07/22/08 07:25:27] TELNET:10.1.20.1 rwa info
Slot5    49 [07/22/08 07:25:26] TELNET:10.1.20.1 rwa nifo
Slot5    48 [07/22/08 07:25:25] TELNET:10.1.20.1 rwa choice secondary
Slot5    47 [07/22/08 07:25:20] TELNET:10.1.20.1 rwa choice
Slot5    46 [07/22/08 07:25:16] TELNET:10.1.20.1 rwa info
Slot5    45 [07/22/08 07:25:13] TELNET:10.1.20.1 rwa choice secondary image-file /flash/p80a4160.img
Slot5    44 [07/22/08 07:24:57] TELNET:10.1.20.1 rwa choice secondary ?
Slot5    43 [07/22/08 07:24:51] TELNET:10.1.20.1 rwa choice
Slot5    42 [07/22/08 07:24:34] TELNET:10.1.20.1 rwa choice image-file ?
Slot5    41 [07/22/08 07:24:25] TELNET:10.1.20.1 rwa choice ?
Slot5    40 [07/22/08 07:24:22] TELNET:10.1.20.1 rwa ?
Slot5    39 [07/22/08 07:24:21] TELNET:10.1.20.1 rwa bootconfig
Slot5    38 [07/22/08 07:24:19] TELNET:10.1.20.1 rwa config
Slot5    37 [07/22/08 07:24:09] TELNET:10.1.20.1 rwa more /flash/boot.cfg
Slot5    36 [07/22/08 07:24:05] TELNET:10.1.20.1 rwa save bootconfig
Slot5    35 [07/22/08 07:24:02] TELNET:10.1.20.1 rwa config bootconfig choice secondary image-file /flash/p80a4160.img
Slot5    34 [07/22/08 07:24:01] TELNET:10.1.20.1 rwa config bootconfig choice primary image-file /flash/p80a4163.img
Slot5    33 [07/22/08 07:22:50] TFTP:127.0.0.6 get /p80a4163.img
Slot5    32 [07/22/08 07:22:36] TELNET:10.1.20.1 rwa peer telnet
Slot5    31 [07/22/08 07:22:33] TELNET:10.1.20.1 rwa mv /flash/p80a4163.mig /flash/p80a4163.img
Slot5    29 [07/22/08 07:22:01] TFTP:127.0.0.6 get /flash/p80a4163.img
Slot5    28 [07/22/08 07:21:43] TELNET:10.1.20.1 rwa peer telnet
Slot5    27 [07/22/08 07:21:42] TELNET:10.1.20.1 rwa mv /flash/p80a6163.img /flash/p804163.img
Slot5    26 [07/22/08 07:21:31] TELNET:10.1.20.1 rwa dir
Slot5    25 [07/22/08 07:21:19] TFTP:127.0.0.6 get /p80j4163.dld
Slot5    24 [07/22/08 07:17:26] TFTP:127.0.0.6 get /p80m4163.img
Slot5    23 [07/22/08 07:17:25] TFTP:127.0.0.6 get /p80c4163.aes
Slot5    22 [07/22/08 07:17:25] TFTP:127.0.0.6 get /p80c4163.img
Slot5    21 [07/22/08 07:17:19] TFTP:127.0.0.6 get /p80b4163.img
Slot5    20 [07/22/08 07:17:19] TFTP:127.0.0.6 get /p80a4163.img
Slot5    19 [07/22/08 07:16:33] TELNET:10.1.20.1 rwa peer telnet
Slot5    18 [07/22/08 07:16:22] TELNET:10.1.20.1 rwa dir
ERS-8610:5#

Cheers!

Recent Posts

Categories

SUBSCRIBE BY EMAIL