I had an interesting issue this past week when I performed a software upgrade on a Gigamon GigaVUE-420. While the upgrade was fairly straight forward I ran into a problem after the upgrade with the IBM Tealeaf solution. We have multiple SPANs and TAPs feeding data into the Gigmon which then copies the traffic out to a number of solutions, including IBM’s Tealeaf. After the upgrade all the other systems seemed to be working fine with the exception of the Tealeaf Linux capture server. The model of Gigamon we have doesn’t allow for altering the actual data, we can filter the data based on anything in the headers but we can’t alter the data. The SPANs from our Cisco 6509E and 6504E switches were setup as 802.1q tagged trunks so the Gigamon would replicate the frames as 802.1q tagged packets. The issue appears to have been how the IBM Tealeaf Linux server handles 802.1q tagged packets. I was able to connect a Windows 7 laptop to the Gigamon and validate that the Gigamon was working properly. I did need to make a registry tweak to the Windows 7 laptop so it wouldn’t strip the 802.1q headers.
Unfortunately IBM support wasn’t very helpful, they were more interested in placing blame than they were in helping us understand why the Tealeaf capture server wasn’t working. They were completely focused on the fact that it worked before the upgrade so it must have been the upgrade that broke it. While that was technically true there was something else at play since I had already verified that the traffic was being forward properly by the Gigamon.
Ultimately one of the team members reconfigured the Linux NICs to support 802.1q tagging and built sub-interfaces so tcpdump could read the traffic. I never did find out what broke but I’m guessing it has something to-do with the NIC configuration on the Tealeaf Linux capture server.
Cheers!
Image Credit: John