Web Application Load Testing – TCP Port Exhaustion

914288_69337190-scale

I recently ran into an puzzling issue with a web framework that was failing to perform under a load test. This web framework was being front-ended by a pair of Cisco ACE 4710 Application Control Engine (Load-Balancer) using a single IP address in a SNAT pool. The Cisco ACE 4710 was the initial suspect, but a quick analysis determined that we were potentially experiencing a TCP port exhaustion issue because the test would start failing almost at the same point every time. While the original suspect was the Cisco ACE 4710 it turned out to be a TCP port exhaustion on the web application tier. The load test was hitting the site so hard and so fast that it was cycling through all ~ 64,000+ possible TCP ports before the web server had freed up the TCP port from the previous request on that same port. The ports were in TIME_WAIT state even though the Cisco ACE 4710 had sent a FIN requesting the port be CLOSED. Thinking the port was available the Cisco ACE 4710 attempted to make a connection on the port a second time which failed because the web application tier still had the TCP port in a TIME_WAIT state and hadn’t closed or freed up the port. While the Linux system administrators attempted to tune their web application tier we still had issues with TCP ports overlapping between requests so the intermin solution was to add 4 more IP addresses to the SNAT pool on the Cisco ACE 4710. This way we’d need to go through 5 * 64,000 TCP ports before we’d need to cycle back through the ports.

References;
LogNormal – http://www.lognormal.com/blog/2012/09/27/linux-tcpip-tuning/

Cheers!

Image Credit: Jaylopez

{ 2 comments }

Response: Scripting Does Not Scale For Network Automation

screencapture-etherealmind-com-scripting-scale-network-automation

About three weeks ago Greg Ferro from Etherealmind posted an article entitled "Scripting Does Not Scale For Network Automation". It's quite clear from reading the article that Greg really is "bitter and jaded".  While I agree that there are challenges in scripting they also come with some large rewards for those that are able to master the skill. In a subsequent comment Greg really hits on his point.. "We need APIs for device consistency, frameworks for validation and common actions. But above that we need platforms that solve big problems - scripting can only solve little problems. " I agree […] Read More

{ 0 comments }

Your customer needs help? Tell them to hire me!

831838_16000623-scale

This is a little off-topic but I've probably let this slide for too long and unfortunately I've been going around with this bent up anger for quite sometime now and it's time to vent and rant. I provide a blog and forum to the community as a way to help educate people and hopefully learn a little something myself along the way. I'm generally interested in targeting the actual end-user, the network engineer or system administrator that's working for Acme Corp. or Wayne Enterprises or the Umbrella Corp, hopefully you get the idea. Inevitably there will be a reseller or […] Read More

{ 1 comment }

CrashPlan filling up your SSD?

crashplan

Over the weekend I actually had some downtime and was hoping to play a little Planetside 2 until I noticed that my Windows 7 desktop was down to only 8GB of free space on my 256GB SSD.  A quick check with WinDirStat found that I had over 133GB of files in C:\ProgramData\CrashPlan, even though I had installed the software into D:\Program Files (x86)\CrashPlan. I've been testing CrashPlan for the past 30 days trying to decide if it was the best tool available for me to use in backing up the numerous desktops and laptops throughout the house. I had been […] Read More

{ 0 comments }

100% Cable Patching in the Closet?

590347_68559935

With the move to a new organization comes some adjustment and occasionally some surprises. While my last employer was very rigid and bureaucratic, my new employer is extremely flexible, deplores 'process for the sake of process' and is extremely fast paced. There are pros and cons to everything in life although I'm not going to talk about my employers today. Instead let's talk about patching in the closet, sometimes referred to as the ICR (Intermediate Communications Room) but more often referred to as the IDF (Intermediate Distribution Frame).  There are a few different mindsets when it comes to patching the […] Read More

{ 2 comments }