Michael McNamara

technology, networking, virtualization and IP telephony

Troubleshooting Opengear via SSH

by

We had an odd problem over the weekend… a recently installed Opengear ACM7004 started intermittently dropping off the internal network, interestingly enough this Opengear was also connected to the public Internet and was having no issues on that NIC, so we had to do some basic troubleshooting from a reverse SSH tunnel – no Web user interface.

I wanted to-do some basic troubleshooting;

  1. Is there LINK?
  2. What speed and duplex are we auto-negotiating?
  3. Any errors on the switch side or host side?

There are a few different tools in Linux to help troubleshoot basic network connectivity issues, ifconfig, netstat, ethtool, and ip are among the top of the pile.

$ ethtool eth0
Settings for eth0:
	Supported ports: [ TP MII ]
	Supported link modes:   10baseT/Half 10baseT/Full 
	                        100baseT/Half 100baseT/Full 
	                        1000baseT/Half 1000baseT/Full 
	Supports auto-negotiation: Yes
	Advertised link modes:  10baseT/Half 10baseT/Full 
	                        100baseT/Half 100baseT/Full 
	                        1000baseT/Half 1000baseT/Full 
	Advertised auto-negotiation: Yes
	Speed: 10Mb/s
	Duplex: Half
	Port: MII
	PHYAD: 0
	Transceiver: external
	Auto-negotiation: on
Cannot get wake-on-lan settings: Operation not permitted
	Link detected: no

In the above output eth0 is down, in the below output eth0 is up. The NIC appeared to be bouncing up and down intermittently on and off for no apparent reason.

$ ip link
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0:  mtu 1500 qdisc mq state UP qlen 532
    link/ether 00:13:c6:aa:bb:cc brd ff:ff:ff:ff:ff:ff
3: eth1:  mtu 1500 qdisc mq state UP qlen 532
    link/ether 00:13:c6:aa:bb:cc brd ff:ff:ff:ff:ff:ff

Ok, so I was specifically interested in eth0 and at the time it was reachable (ie. working) so I had a look at the Juniper EX4300 switch and found just a few issues;

 show interfaces ge-1/0/4 extensive | match error 
   Link-level type: Ethernet, MTU: 1514, MRU: 0, Speed: Auto, Duplex: Auto, BPDU Error: None, MAC-REWRITE Error: None, Loopback: Disabled,
   Input errors:
     Errors: 10, Drops: 0, Framing errors: 10, Runts: 0, Policed discards: 0, L3 incompletes: 0, L2 channel errors: 0, L2 mismatch timeouts: 0,
     FIFO errors: 0, Resource errors: 0
   Output errors:
     Carrier transitions: 1399, Errors: 0, Drops: 0, Collisions: 0, Aged packets: 0, FIFO errors: 0, HS link CRC errors: 0, MTU errors: 0,
     Resource errors: 0
     CRC/Align errors                        10                0
     FIFO errors                              0                0

There were 1399 carrier transitions (the port had bounced up and down 1399 times). So that immediately told me there was definitely a problem somewhere. The CRC/Align errors could be a result of the port bouncing so much. I was able to quickly correlate the logs from the Juniper switch to the monitoring system. The monitoring system was loosing connectivity to the Opengear when the switch port was going down – which is obviously expected. So this was essentially a physical Layer1 problem – perhaps a cabling issue?

1000BaseT requires all 8 wires in order to make a connection, 100BaseT only requires 4 wires to make a connection, so I changed the Juniper switch to auto-negotiate at 10Mbps or 100MBps, and not 1Gbps and the port immediately connected.

set interfaces ge-1/0/4 speed auto-10m-100m

I’m going to guess that we have a bad patch cable between the Juniper EX4300 and the Opengear ACM7004, but for now we can run the Opengear at 100Mbps without an issue.

Cheers!

Opengear pmshell ESCAPE sequences

by

If your in pmshell you’ll need to use the following escape sequences to get back to the “Connect to port >” prompt if for example you want to connect to different serial port. If you weren’t confused enough, if you are connecting via SSH you’ll need to use an extra ~ before those escape sequences so the ~ isn’t interpreted by your SSH client, it needs to pass through to pmshell.

Shell Commands:
~b - Generate BREAK
~h - View history
~p - Power menu
~m - Connect to port menu
~. - Exit pmshell
~? - Show this message

I’ve noticed on occasion that I need to be fairly quick with the keyboard strokes or pmshell will just pass the characters through to the device connected to the serial port.

Cheers!
Image Credit to Michaelaw

Opengear Console Servers – Never leave the office without one!

by

I’m continually amazed by how many large enterprise organizations don’t have out-of-band access to the critical pieces of their infrastructure. I’m speaking about the console ports of key routers, switches, firewalls, application switches, etc in their network infrastructure and iLO (HP) or RSA (IBM) for their critical server infrastructure. In this day and age of 24 x 7 x 365 up-time, waiting any amount of time for folks to physically respond is just plain wasteful and counter productive. There are occasions where you’ll still need to physically respond but the troubleshooting process can be exponentially expedited with out-of-band access to the actual equipment. I’m currently working in healthcare so up time and reliability are critical to our day to day operations, and more so to the actual health and well being of our patients and customers.

We count downtime in minutes and seconds not hours or days and so we need to ensure that we have access to the necessary tools to maintain the infrastructure. I’ve personally gone years without needing console access to a few specific sites but that once or twice that I quickly needed access easily justified the capital and operating costs of having the solution available. It’s not difficult to design highly available and redundant network, server and facility infrastructures. However, even with all that redundancy you can find yourself desperately seeking that console access at the most inopportune times (2:00AM comes to mind).

ACM5508-2-M-2I’ve really taken a liking to the Opengear products. They have multiple models including 1, 3, 4, 8, 16 and 48 port options and offer access via Ethernet, Wi-Fi, 3G, 4G LTE and v.92 modem. You can actually send the 3G unit an SMS text message to wake it up, so it’s not constantly online drawing needlessly against your monthly 3G bandwidth allocation, and they all support Dynamic DNS registration so you don’t need a static public IP address. The latest models also support USB, so you can connect to those Cisco USB console ports.

Here are my notes for configuring an ACM5003-W, the model we’re deploying in our edge closets. I have yet to figure out how to make use of the wireless interface as a back-up to the LAN interface. Apparently the current software doesn’t support that configuration but hopefully that will change in the future.

You’ll be able to connect to the Opengear console server on port 1 using 115200-8-N-1.

The default username is root while the default password is default.

Once you are logged in the following commands will configure the Opengear via the CLI interface. You can also launch the web interface to verify your configuration at a later point in time.

We start the process by configuring the name and location of the console server. The last command below will prompt you to change the root password – I’ve had issues where the change doesn’t seem to be working.

config -s config.system.name=ts-og1-rack2.testlab.acme.org
config -s "config.system.location=ACME Testlab (Rack 2)"
config -P config.system.password

Let’s set the IP addressing. If we were using the wireless LAN interface as opposed to the Ethernet interface any references below to wan would be replaced with wlan.

config -s config.interfaces.wan.address=10.1.255.11
config -s config.interfaces.wan.netmask=255.255.255.0
config -s config.interfaces.wan.gateway=10.1.255.1
config -s config.interfaces.wan.dns1=10.1.16.40
config -s config.interfaces.wan.dns2=10.7.16.40
config -s config.interfaces.wan.mode=static
config -s config.interfaces.wan.media=Auto

Let’s enable NTP and set the date and time on the device;

config -s config.ntp.enabled=on
config -s config.ntp.server=tick.acme.org
config -s config.system.timezone=US/Eastern

Let’s set the date and time to something close, NTP will give us the precision we want.

# date Format is MMDDhhmm[[CC]YY][.ss]
date 121009472013.00
/bin/hwclock --systohc

Let’s set the serial ports. In this case we will skip port 1, the port we are using it to configure the device. If we need all three ports we can configure it but the changes don’t take effect until we issue a “config –a”.

# This is the port we are currently using to configure the Opengear so let's skip it right now.
#config -s config.ports.port1.speed=9600
#config -s config.ports.port1.parity=None
#config -s config.ports.port1.charsize=8
#config -s config.ports.port1.stop=1
#config -s config.ports.port1.label=’Other Device’
#config -s config.ports.port1.loglevel=0
#config -s config.ports.port1.protocol=RS232
#config -s config.ports.port1.flowcontrol=None
#config -s config.ports.port2.mode=portmanager
#config -s config.ports.port2.ssh=on
#config -s config.ports.port2.telnet=on
#config -s config.ports.port2.webshell=on

Let’s configure serial port 2 and 3;

config -s config.ports.port2.speed=9600
config -s config.ports.port2.parity=None
config -s config.ports.port2.charsize=8
config -s config.ports.port2.stop=1
config -s config.ports.port2.label='Juniper SRX210H'
config -s config.ports.port2.loglevel=0
config -s config.ports.port2.protocol=RS232
config -s config.ports.port2.flowcontrol=None
config -s config.ports.port2.mode=portmanager
config -s config.ports.port2.ssh=on
config -s config.ports.port2.telnet=on
config -s config.ports.port2.webshell=on

config -s config.ports.port3.speed=9600
config -s config.ports.port3.parity=None
config -s config.ports.port3.charsize=8
config -s config.ports.port3.stop=1
config -s config.ports.port3.label='Other Device'
config -s config.ports.port3.loglevel=0
config -s config.ports.port3.protocol=RS232
config -s config.ports.port3.flowcontrol=None
config -s config.ports.port3.mode=portmanager
config -s config.ports.port3.ssh=on
config -s config.ports.port3.telnet=on
config -s config.ports.port3.webshell=on

Let’s create a user account called admin and set the proper privledges;

config -s config.users.total=2 (assuming we already have 1 user configured)
config -s config.users.user2.username=admin
config -s config.users.user2.description=AdministrativeUser
config -s config.users.user2.port1=on
config -s config.users.user2.port2=on
config -s config.users.user2.port3=on
config -s config.users.user2.groups.group1='admin'
config -s config.users.user2.groups.group2='dialin'
config -s config.users.user2.groups.group3='pmshell'
config -s config.users.user2.groups.group4='ftp'
config -s config.users.user2.groups.group5='users'
config -s config.users.user2.groups.group6='pptpd'
config -P config.users.user2.password

Let’s set the SMTP settings, not sure that we will really use them though;

config -s config.system.smtp.server=smtp.acme.org
config -s config.system.smtp.encryption=none
config -s config.system.smtp.sender=opengear@acme.org
config -s config.system.smtp.subject=Opengear Alerts

Let’s set the SNMP settings;

config -s config.system.snmp.protocol=UDP
config -s config.system.snmp.trapport='162'
config -s config.services.snmp.syslocation='ACME Testlab (Rack 2)'
config -s config.services.snmp.syscontact='ACME Technology Team'
config -s config.system.snmp.address='10.1.1.125'
config -s config.system.snmp.community='public'
config -s config.services.snmp.rocommunity='public'
config -s config.services.snmp.rwcommunity='private'
config -s config.system.snmp.version=2c

Let’s enable the various system services;

config -s config.services.http.enabled=on
config -s config.services.https.enabled=on
config -s config.services.telnet.enabled=on
config -s config.services.ssh.enabled=on
config -s config.services.snmp.enabled=on
config -s config.services.pingreply.enabled=on
config -s config.services.tftp.enabled=on
config -s config.services.webshell.enabled=on

Let’s enable the modem (if we have one);

config -s config.modem.ddns.provider=none
config -s config.modem.flow=Hardware
config -s config.modem.ppp.enabled=no
config -s config.modem.speed=9600

The last step we must take is to activate and sync all the changes to flash memory.

config -a

PLEASE MAKE SURE YOU DIDN’T SKIP THE STEP ABOVE – VERY IMPORTANT

When you are at the $ or # shell prompt you can launch the Port Manager Shell with the command;
pmshell

You should be give a list of ports you can connect to such as the following;

# pmshell
2: Juniper SRX210H 3: Other Device
Connect to port >

You only need to enter the number for the selection you would like.
The break command is ~. or ~? for help

If you don’t have any out-of-band access I’d suggest that you make that a new year resolution!

Cheers!

Opengear

by

I met with Opengear on the sixth floor sixth floor of Casino M8trix in San Jose, CA on Wednesday afternoon October 10, 2012 for Networking Field Day 4. The Networking Field Day 4 delegates and myself specifically met with Jared Mallet and Toby Smith of Opengear.

I have no personal experience with Opengear although I’ve used competing products to provide out-of-band serial/console access in my data centers and large computer rooms. I can’t remember who but someone referred to these devices as the “save my bacon” device and I would definitely agree. I can’t begin to state the value of having an out-of-band network available once you start having problems or even when performing upgrades or scheduled maintenance. They have literally helped me resolve problems in minutes as opposed to hours. Anyone that’s serious about their network should have every serial/console port wired to a console/terminal server for out-of-band access via dial-up or as you’ll hear in the Tech Field Day videos cellular 3G access.

I’m going to outline the different presentations that we heard and perhaps make a few points here and there if I have anything useful to say. I’ll include a short blurb from OpenGear in italics to help define/describe each product or discussion point. Thankfully since the sessions were recorded you can watch for yourself and make your own informed opinion.

Here’s my disclaimer; I’m not endorsing any of the solutions presented below. I’m merely passing on the information along with a few comments of my own. If you have any personal opinions about the solutions below why not share them with us in the comments?

OpenGear Products

Tech Field Day Video
by Jared Mallett and Toby Smith

Opengear is a leading provider of enterprise grade console servers, cellular routers and remote management solutions that enable our customers to easily tame IT complexity. Our commitment to simplify remote management enables our products to deliver what IT professionals really need and want with unquestionable security and value.

Opengear’s extensive support for out-of-band management ensures our customers have access and control even when their network is down. The result is a drastic reduction in the need for expensive on-site technical staff, on-site visits and rapid problem resolution.

  • DCIM (Data Center Infrastructure Management)
  • RIM (Remote Infrastructure Manager)

The rack mount appliances include the following models; CM4000, IM4200, IM4216-34 – combines both IP and serial access saving rack space.

My Thoughts?

This device is the swiss army knife of the data center engineer. It truly is the life line to your data center allowing you to access the serial/console port of your switch, router or appliance in the critical minutes during an unscheduled outage allowing you to quickly and remotely troubleshoot and resolve the problem or issue. The appliances run a variant of uClinux and support remote access via PPP (modem), PPtP, OpenVPN, and IPSec. OpenGear provides free software downloads for the life of the product, no maintenance contract required.

I was provided an 4 port console server model ACM50004-G by Opengear which I’ll be drafting a review of in a future post.

VCMS – OpenGear version of Nagios

Tech Field Day Video
by Jared Mallett and Toby Smith

Opengear VCMS is a powerful system that centrally monitors and manages distributed Opengear products and their managed devices from a single console. VCMS delivers flexible target device management control for large data centers with a hundred rows of racks, or in power utility networks with thousands of smart appliances. Designed to reduce downtime and establish a single point of access, the VCMS aggregates “call home” sessions from isolated devices initiated remotely both in-band and out-of-band. The VCMS allows for unified visibility of all managed devices across large enterprise networks seamlessly from a common cloud hosted appliance with intuitive browser interface.

PDU/RPC Power Managment – PowerMan

My Thoughts?

If you happen to have a few dozen console servers hanging around your environment you can leverage VCMS to help manage them. If you are already running Nagios, OpenGear supports running NRPE and the NSCA Nagios client checks locally on the actual ACM, CM, IM console servers since they are running uClinux. I’m excited to see OpenGear leveraging other open source projects such as Nagios to help their customers manage their appliances.

Cellular Out-of-band Management

Tech Field Day Video
by Jared Mallett and Toby Smith

Managing distributed IT infrastructure is hard enough. Why make it more complex and expensive by having to buy, deploy and manage multi-vendor proprietary management tools? An integrated out-of-band management solution should be a flexible solution that deploys quickly, begins working immediately, is simple to use and manage, and integrates seamlessly with existing IT management systems.

The Opengear ACM5000 and IM4200-X2 families support internal and external cellular modems. These modems will need to be provisioned by the cellular carrier for a data plan. Once provisioned, the Opengear devices can than be configured to operate in variety of modes for cellular connectivity.

My Thoughts?

Now you can access your console server via a cellular 3G network as opposed to a out-of-band network or dial-up modem. The Opengear appliances support DyDNS for dyanmic IP address assignment and even support call home functionality over IPSec or OpenVPN to a centralized management solution such as VCMS. Static IP address assignment is also available if you can get the carrier to provide one. The console server also support SMS text messaging as a means of waking it up, so your not using up your mobile minutes or bandwidth allocation until you actually need the device.

Demo – Advanced Console Management via Web Dashboard (Tech Field Day Video)

Closing

I’d like to thank Opengear along with Jared and Toby, we had a great discussion around a very important piece of data center equipment. If you are relying on your data center providers “Smart Hands” I’ll warn you now, the cost of these devices will easily pale in comparison to the costs you’ll incur when you can’t get access to troubleshoot an outage. I’ll be personally evaluating Opengear against my incumbent vendor.

Cheers!

Recent Posts

Categories

SUBSCRIBE BY EMAIL