Michael McNamara https://blog.michaelfmcnamara.com technology, networking, virtualization and IP telephony Sat, 09 Jan 2016 14:44:54 +0000 en-US hourly 1 https://wordpress.org/?v=6.7.2 Network Troubleshooting and Wireshark https://blog.michaelfmcnamara.com/2016/01/network-troubleshooting-and-wireshark/ Sat, 09 Jan 2016 14:43:21 +0000 http://blog.michaelfmcnamara.com/?p=5489 In a recent troubleshooting session with an Avaya IP Office system we had to perform packet traces from both an Avaya IP Office server using tcpdump and from an Avaya IP Office gateway using a port mirror on an Avaya 3500 series switch. The topology was a pretty simple flat network with only 2 switches and 2 VLANs (data and voice). The vendor had asked for some packet traces from both the Avaya IP Office gateway and from the Avaya IP Office server. We were able to obtain the data the next step was to analyze the data… how to make sense of all the noise in the packet trace and discern exactly what was going on. And what (if any) conclusions could be drawn from the collected data.

I thought I would share my default Wireshark setup I use when examining packet traces. In addition to the defaults I like having the Time (since capture start), the DateTime, (absolute time is useful when correlating against other packet traces and log files), DeltaX (time between displayed packets), Seq, Ack, and Bytes in flight.

Wireshark-Columns

I have a color rule which highlights the frames in yellow if the frame.time_delta_diplayed is greater than 3 seconds (frame.time_delta_displayed > 3). It helps me to quickly focus on long pauses in communication between two or more devices. While working on this problem I also happened to stumble upon a bug in Wireshark 2.0 that affects the delta time displayed. I discovered bug 11786 was already documented in the Wireshark bug database.

If you’re not familiar with how to read packet traces I would suggest you check out Kary Rogers Packet Bomb website. Karry has a number of useful videos covering how to read and interpret a packet trace, along with a few tips on leveraging tools like Wireshark. In the screenshot above I used Jasper Bongertz’s tool TraceWrangler to sanitize the IP addresses from the packet trace before post a screenshot.

Cheers!

Update: The delta time issue has been fixed in Wireshark 2.0.1

You might also want to read these other posts...

  1. UNISTIM Protocol (WireShark)
  2. Remote Packet Capture with WireShark and WinPCAP
  3. Wireless Packet Traces (AirPcap)

]]> IBM Tealeaf – Gigamon 802.1q Tagged Packets https://blog.michaelfmcnamara.com/2015/03/ibm-tealeaf-gigamon-802-1q-tagged-packets/ Sun, 01 Mar 2015 17:52:01 +0000 http://blog.michaelfmcnamara.com/?p=5264 I had an interesting issue this past week when I performed a software upgrade on a Gigamon GigaVUE-420. While the upgrade was fairly straight forward I ran into a problem after the upgrade with the IBM Tealeaf solution. We have multiple SPANs and TAPs feeding data into the Gigmon which then copies the traffic out to a number of solutions, including IBM’s Tealeaf. After the upgrade all the other systems seemed to be working fine with the exception of the Tealeaf Linux capture server. The model of Gigamon we have doesn’t allow for altering the actual data, we can filter the data based on anything in the headers but we can’t alter the data. The SPANs from our Cisco 6509E and 6504E switches were setup as 802.1q tagged trunks so the Gigamon would replicate the frames as 802.1q tagged packets. The issue appears to have been how the IBM Tealeaf Linux server handles 802.1q tagged packets. I was able to connect a Windows 7 laptop to the Gigamon and validate that the Gigamon was working properly. I did need to make a registry tweak to the Windows 7 laptop so it wouldn’t strip the 802.1q headers.

Unfortunately IBM support wasn’t very helpful, they were more interested in placing blame than they were in helping us understand why the Tealeaf capture server wasn’t working. They were completely focused on the fact that it worked before the upgrade so it must have been the upgrade that broke it. While that was technically true there was something else at play since I had already verified that the traffic was being forward properly by the Gigamon.

Ultimately one of the team members reconfigured the Linux NICs to support 802.1q tagging and built sub-interfaces so tcpdump could read the traffic. I never did find out what broke but I’m guessing it has something to-do with the NIC configuration on the Tealeaf Linux capture server.

Cheers!

Image Credit: John

You might also want to read these other posts...

  1. untagAll vs tagAll on Avaya Ethernet Routing Switches
  2. Packet Capture on a Blue Coat ProxySG Appliance
  3. Avaya VPN Client Release 10.06.200 for Windows 8

]]> Remote Packet Capture with WireShark and WinPCAP https://blog.michaelfmcnamara.com/2010/09/remote-packet-capture-with-wireshark-and-winpcap/ https://blog.michaelfmcnamara.com/2010/09/remote-packet-capture-with-wireshark-and-winpcap/#comments Sun, 05 Sep 2010 14:22:26 +0000 http://blog.michaelfmcnamara.com/?p=1619 I’m just continually impressed with the quality of so many open source products available today. One such product that should be extremely high on any network engineer’s list is WireShark. WireShark has become the de-facto standard for packet capture software and is almost unrivaled in features and functionality.

Last week I had the task of diagnosing some very intermittent desktop/application performance issues at a remote site. I had installed WireShark locally on a few desktops but I wanted the ability to remotely monitor a few specific desktops without obstructing the users workflow to get a baseline for later comparison. I was excited to learn that WireShark and WinPCAP had (experimental) remote packet capture functionality built into each product. I followed the instructions on the WireShark website by installing WinPCAP v4.1.2 on the remote machine and then starting the “Remote Packet Capture Protocol v.0 (experimental)” service. With that done I then proceeded to launch WireShark on my local desktop and configure the remote packet capture settings. From within WireShark I chose Options -> Capture, changed the Interface from Local to Remote. Then enter the IP address of the remote machine along with the TCP port (the default TCP port is 2002). I initially tried to use “Null authentication” but was unsuccessful. I eventually ended up choosing “Password authentication” and used the local Administrator account and password of the remote desktop that had WinPCAP installed on it. If the remote desktop had multiple interfaces I could have selected which interface I wanted to perform the remote packet capture on. In this case the desktop in question only had an integrated Intel(R) 82567LM-3 network adapter. I clicked ‘Start’ and to my sheer amazement the packet trace was off and running collecting packets from the remote desktop. There will still be the occasional need to place the Dolch (portable sniffer) onsite when the situation demands it  but this is a great tool to have available.

Cheers!

Updated: Sunday September 5, 2010
The images appear to be missing above because the URL paths are wrong, not sure how WordPress messed up that. I don’t have time right now to fix it but I will fix it a little later.

You might also want to read these other posts...

  1. Packet Capture on a Blue Coat ProxySG Appliance
  2. Packet Capture (PCAP)
  3. Wireless Packet Traces (AirPcap)

]]> https://blog.michaelfmcnamara.com/2010/09/remote-packet-capture-with-wireshark-and-winpcap/feed/ 2