Michael McNamara

technology, networking, virtualization and IP telephony

Network Time Protocol (NTP)

by

1111976_16319994 [ad name=”ad-articlebodysq”]I’m sometimes amazed at how many large organizations don’t have a centralized Network Time Protocol (NTP) server setup and devices configured appropriately. When troubleshooting a problem it’s vital that the timestamps in the logs for each switch, router, server and appliance match up correctly.

I’m currently using two CentOS Linux servers to provide time services to over 10,000 devices in the network. My two servers are themselves syncing up with pool.ntp.org over the Internet. With CentOS I didn’t need to build the software, I only needed to install the NTP package through YUM and then configure it appropriately. It was really easy, much easier than it was say 10 years ago when you had to compile the NTP software (University of Delaware) by hand hoping you didn’t run into some missing library of version mismatch with the compiler.

We would first need to install the NTP software using YUM;
[root@hostname ]# yum install ntp

We would need to start the NTP daemons;
[root@hostname ]# service ntpd start

We would need to configure the server so the NTP software would start after every reboot;
[root@hostname ]# chkconfig ntpd on

With that step done we’d have ourselves and internal NTP server which would sync itself to the Internet (default configuration file in /etc/ntp.conf) and then our internal devices would sync to it.

Here are the CLI commands for configuring the ERS 8600 switch properly;

config bootconfig tz dst-name "EDT"
config bootconfig tz name "EST"
config bootconfig tz offset-from-utc 300
config bootconfig tz dst-end M11.1.0/0200
config bootconfig tz dst-start M3.2.0/0200

config ntp server create a.b.c.d
config ntp server create a.b.c.d
config ntp server create a.b.c.d
config ntp enable true

I’ve add the two configuration statements for the new Daylight Saving Time changes that were enacted in 2007. Please also note that I’m in the Eastern timezone (EDT/EST) so if you’re not in the Eastern timezone you would need to supplement your timezone abbreviation appropriately.

Here are the commands for an ES460,ES470,ERS4500 or ERS5500 series switch

5520-48T-PWR# config terminal
5520-48T-PWR (config)# sntp server primary a.b.c.d
5520-48T-PWR (config)# sntp server secondary a.b.c.d
5520-48T-PWR (config)# sntp enable
5520-48T-PWR (config)# exit5520-48T-PWR#

The ERS 4500/5500 Series now supports Daylight Saving Time. This feature is NOT supported on the ES460 and ES470 switches. –-CORRECTION: this feature is support on the ES460/470 as of v3.7.x software, please see update at the bottom of this post for additional information. If you wanted to configure the timezone on the ERS4500/ERS5500 switch you would use the following commands;

5520-48T-PWR>enable
5520-48T-PWR# config terminal
5520-48T-PWR (config)# clock time-zone EST -5
5520-48T-PWR (config)# clock summer-time EDT date 9 Mar 2008 2:00 2 Nov 2008 2:00 +60
5520-48T-PWR (config)# exit
5520-48T-PWR#

You can use “show sntp” and “show clock” the ERS 5500 Series switch to check out your changes;

5530-24TFD#show sntp
SNTP Status:                      Enabled
Primary server address:         10.1.20.1
Secondary server address:     10.1.20.1
Sync interval:                      24 hours
Last sync source:                 10.1.20.1
Primary server sync failures:    0
Secondary server sync failures: 0
Last sync time:                  2008-06-14 14:47:31 GMT-04:00
Next sync time:                  2008-06-15 14:47:31 GMT-04:00
Current time:                     2008-06-15 13:52:24 GMT-04:00

5530-24TFD#show clock
Current SNTP time  :    2008-06-15 13:52:29 GMT-04:00
Summer time is set to:
start: 28 March 2007 at 02:00
end: 30 August 2008 at 15:00
Offset: 60 minutes. Timezone will be 'EDT'Time Zone is set to 'EST', offset from UTC is -05:00

Hopefully this will provide a brief look into NTP,SNTP and you’ll agree that it really isn’t that hard to setup and configure properly.

Cheers!

Update: June 17, 2008

After posting the article above I decided I would confirm that the Daylight Saving Time feature was not available on the Nortel Ethernet Switch 460/470. I found that as of v3.7.x software the feature is supported on the switches. The configuration commands are identical to the ERS4500/ERS5500 switches. Here’s an example specifically for the Eastern timezone.

470-48T>enable470-48T#config term
Enter configuration commands, one per line.  End with CNTL/Z.
470-48T(config)#clock time-zone EST -5 00
470-48T(config)#clock summer-time EDT date 9 Mar 2008 02:00 2 Nov 2008 2:00 +60
470-48T(config)#show clock summer-time
Summer time is set to:start: 9 March 2008 at 02:00end: 2 November 2008 at 02:00
Offset: 60 minutes. Timezone will be 'EDT'
470-48T(config)#exit

Cheers!

Multi Router Traffic Grapher & RRD

by

Mrtg_logo2 I recently needed to share some network utilization data with some non IT folks in our organization. I produced a quick report from a dynamic HTML page that contained multiple MRTG graphs. Needless to say the graphs did most of the talking while I just answered the questions. One person commented that they didn’t know we had purchased such an elaborate management and monitoring solution. In short we hadn’t purchased any high-end management or monitoring solution, but we had setup Multi Router Traffic Grapher (MRTG) and Round Robin Database (RRD) both of which were written by Tobi Oetiker with contributions from many others.

I’ve been personally using MRTG for well over 10 years now and I’ve yet to find any product (commercial or open source) that comes close. These two tools work hand in hand to help me graph and chart almost any SNMP value (you can also graph non-SNMP values but you need a script or something to collect the values) on almost any device connected to the network. The obvious examples for network engineers and architects such as myself is to use MRTG/RRD to help monitor current network utilization and forecast future growth. There are other examples such as graphing the temperature of a computer room or even the amount of rainfall. There are literally hundreds of examples but I’ll leave you to enjoy reading about them all from the MRTG web site.

Here are two quick examples;

Internet Link with XO (Ethernet ~ 50Mbps)
14allCA2K8OQA

Internet Link with Level3 (Ethernet ~ 50Mbps)
14allCAQDNA38

In the above figures MRTG is graphing the average of ifInOctets and ifOutOctects over a 5 minute interval. As I said above you could graph almost any value you wished.

I also use MRTS by Thor Dreier to help get an idea of how much actually data is traversing a specific network or interface. When we recently installed an HP MAS (Medical Archive Solution) which was built around grid computing and virtual storage technologies we observed a 300% increase in WAN traffic as the MAS was replicating data for business continuity purposes.

I will admit that MRTG can be somewhat complicated for the fledgling network engineer. However, there are dozens of implementation guides now available on the MRTG web site, including support for running MRTG on Windows.

Cheers!

Nortel Business Secure Router 222

by

We recently started looking for a more cost effective VPN router for small office and home office environments. With the current price of gas over $4.13/gallon there are a lot of businesses looking to try and ease the strain by effectively utilizing telecommuting for both voice and data applications. In my next few posts I’m going to look at some different technologies that a telecommuter could potential use in the virtual office.

We’re currently using the Nortel VPN Router 1010, 1050 and 1100 models for mid-size to large offices but needed a more cost effective solution for home office environments such as remote call center agents and other professionals. It also doesn’t help that Nortel has manufacture discontinued the 1010, 1050 and 1100 models (the bulletin from Nortel can be viewed here). There are two approaches that we are currently looking at with respect to the remote call center agents; 1) hardware solution with VPN router and IP phone; 2) software solution with VPN client and IP softphone. In this post I’m going to discuss my impressions of the Nortel Business Secure Router 222.

Let me be honest up front and tell you that I’m no fan of the Nortel VPN 200 Series Router from which this product was born. I know from opening a Nortel VPN 221 Router that it appears as if Nortel has OEM the product from Zyxel. I’m not sure if that’s still the case but the GUI of the BSR 222 looks almost identical to the VPN 221.

bsr_222_1000x689“The Business Secure Router 222, specifically designed for the small to medium business (SMB), is a converged broadband access router that provides a secure connection to the Internet via digital subscriber line (DSL) or cable modem broadband services. The Business Secure Router 222 is an advanced, feature-rich router offered at an affordable price.”

We tested the BSR 222 and were very happy with the results. We provisioned multiple IPSec tunnels with Triple DES encryption to a Nortel VPN Router 1700 (V06_05.140) using Asymmetric Branch Office Tunnel (ABOT) in Aggressive mode. In our previous tests with the VPN 221 router we had all sorts of issues with the IPSec tunnels staying up in Aggressive mode. With the BSR 222 we had no such issues using the exact same profile on the VPN Router 1700 we used for the VPN 221.

We also tested connecting a Nortel i2002 over the BSR 222 and found the call quality to be excellent. While I could have paired a BES 50 with the BSR 222 to provide PoE I decided to just use a power supply on the i2002. The hardware solution seems to be a very reliable and stable solution as it probably should be. I would probably guess that a hardware solution such as this would probably cost around $800 (IP ISM, IP Phone, BSR 222). Please just remember that any VPN solution is only as stable as your broadband connection to the Internet.

The default username is “nnadmin” and the default password is “PlsChgMe!”. The default IP address is 192.168.1.1 and the router can be configured from a web browser by using the URL http://192.168.1.1.

In defense of the VPN 221 router it does support a feature called “Control Ping”. When this feature was configured it allowed the VPN 221 to determine if an IPSec tunnel had become disconnected from the far side. It did this by pinging an IP address that was within the tunnel network range. If the ping failed the router would essentially restart the tunnel by disconnecting it and reconnecting it. It would also keep the tunnel active on the far side preventing any keepalive issues from arising. When I configured this feature on the VPN 221 the tunnels seemed to work flawlessly. This same feature is available on the BSR 222 and it may be required if you find your tunnels bouncing up and down.

Cheers!

ERS 8648GTR duplex mismatch

by

8648GTRI came across a bulletin from Nortel just recently that I thought was important enough to post here in case anyone reading this has 8648GTRs installed in his/her ERS 8600 chassis. I have about 10 of them installed at multiple locations, primarily in core switches feeding large server farms and other high-speed devices. I don’t believe I need to extol the pains that auto-negotiation sometimes reaps on network engineers. While modern network switches and NICs are definitely more compatible with respect to auto-negotiation, problems sometimes still arise. It would seem that a duplex mismatch on one port of the 8648GTR could potentially impact performance on up to 24 ports. Here’s the text from the bulletin;

Background:

An Ethernet port can operate either in Full or Half Duplex mode. A duplex mismatch is created when using inconsistent settings for duplex mode, i.e. full duplex on the port and half duplex on the connected device (or vise versa). This situation is most likely created when using inconsistent and inappropriate settings for auto-negotiation, i.e. auto-negotiation enabled on the port and disabled on the device connected to the port (or vise versa). The duplex mismatch problem can be corrected by setting consistent duplex mode on both the port and the connected device when hard setting the duplex mode or by enabling auto-negotiation on both the port and the connected device, when using auto-negotiation.

Ethernet ports of most devices today have auto-negotiation enabled as the default setting. When a device with auto-negotiation disabled is connected to a port that has auto-negotiation enabled, the port is not able to detect the duplex setting of the connected device and falls back to half duplex thus potentially causing a duplex mismatch. A duplex mismatch will cause physical layer errors and performance degradation of the connection. Any mixture of auto-negotiation enabled on one-side and auto-negotiation disabled on the other side is an “unsupported” configuration. The setting on both sides of any connection must match for proper operation. A problem has been identified when there is a duplex mismatch on one or more ports of an 8648 GTR module. For an 8648 GTR module, a duplex mismatch may cause complete communication issues on the port with the mis-matched duplex or occasionally on the entire lane (Port 1-24 or Port 25-48) that contains the port with mismatched duplex. The module can be recovered from the situation when physically reseated, but for complete recovery the mis-configuration must also be corrected. Correcting the duplex setting configuration alone will not recover the communication loss until the module is reseated as well.

Analysis:

A duplex mismatch may cause communication loss on a port or an entire lane of an 8648 GTR module. When there is such a communication loss, the debugging commands show that the ingress stats look normal with all traffic ingressing the impacted port(s) and the MAC addresses learned in the Forwarding Database Table for the devices connected to the port(s), but no traffic egressing the port(s).

Recommendations:

Nortel recommends proper configuration of auto-negotiation whenever possible to prevent a duplex mismatch situation. To avoid a duplex mismatch, auto-negotiation must be enabled on the port as well as the device connected to that port.

You can find a copy of the bulletin in PDF format right here. Interestingly there are quite a few restrictions and issues with the 8648GTR that I should probably discuss them here when the time allows.

Nortel is also in the process of releasing v5.0 software for the Ethernet Routing Switch 8600 along with four new IO modules (cards); 8612XLRS, 8634XGRS, 8648GBRS and 8648GTRS. I hope to talk about those in the very near future.

Cheers!

How much uptime is too much?

by

A quick story for everyone…

We generally perform software upgrades on all our routers and switches twice a year. It really helps to keep our network infrastructure current and it also helps to reduced unscheduled downtime.

Last fall we decided to skip the bi-yearly maintenance because there were just too many projects on the docket. This spring we came across a very interesting issue that we had never seen in the past. We started to notice that multiple Nortel Ethernet Switch 460/470 switches/stacks were rebooting themselves all over our network. It took us a few hours to realize that every switch that had rebooted had just eclipsed approximately 500 days of uptime. All the affected switches were running FW 3.6.0.6 with SW v3.6.4.08. The switches were literally rebooting themselves in the same order in which they had been upgraded almost 500 days earlier.

I’m currently trying to confirm with Nortel that this “bug” has been removed from the 3.7.x software release.

This was one occasion where the network was just too good for itself.

Cheers!

Update: Tuesday June 10, 2008

I received a formal response from Nortel today that included the following:

Analysis of the issue :-
When the BS-470 switches reaches 497 days the system time rolls over and during this period management communication will be lost. This is caused by the use of a 32 bit counter, which when it rolls back to 0, initiates an internal software synchronization to align all timers. This is only loss of IP management and not switching functionality.

This issue still open and can be fixed by rebooting the switches before reaching the 497 day mark.

When I inquired if the problem had been resolved in the v3.7.x software release I was told it had not. It would seem that a lot of folks just don’t expect switches to be running that long these days.

Cheers!

Update: Wednesday November 4, 2008

Last week Nortel released a technical service bulletin entitled, “Ethernet Routing Switches: SysUpTime approaching 497 days can cause the switch or stack to behave in some unexpected way“. They also released a video that documents a workaround to the problem.

Let me save you the time and effort of downloading either. Nortel solution is truely masterful; reboot the switch.

While I’ve been know to defend Nortel there’s just no defense for this. I’m completely floored at Nortel’s response.

Cheers!

Recent Posts

Categories

SUBSCRIBE BY EMAIL