Michael McNamara

technology, networking, virtualization and IP telephony

Response: Is It Really Always The Network? #ITNF

by

I recently read Tom Hollingsworth’s post titled “Is It Really Always The Network?” and immediately thought, I need to find time to post a reply. I’ve been in this industry for more than 20 years now and it has been a constant struggle to educate and train those around me to perform their due diligence before issuing the knee-jerk response that “it must be the network.” If you don’t understand the problem then ask for help, don’t pretend to understand the problem and then assign blame when you have no idea what you are talking about. I suspect people are getting worse and not better, although I almost wonder if people are getting lazier and just want someone else to fix their problems.  Last week I had two examples of people throwing sh*t over the wall without the even performing the most basic troubleshooting steps and as you can already guess I was pissed.

In the first example, a SOAP/XML interface to a third-party wasn’t accepting transactions. Well that’s proof positive that it’s a network problem, right? A simple “telnet” test to the IP address and port from the origin server was successful. Yet the response from the team reporting the problem? “If it’s not the network we don’t know what to-do next” which left me speechless.  After a 15 minute conference call, I had the third-party restart their back-end service and magically transactions started flowing again.

In another example, I was notified that a SOAP/XML interface that we host on the public Internet was inaccessible – must be a network issue. I verified that I was unable to connect to that specific host from the internal network on the TCP port specified and advised that they should check the specific host in question, I was rebuked by the application analyst telling me, “Nginx is up and running”. A co-worker remotely connected to the specific server and found Nginx prompting for the passphrase (password) for the SSL certificate that Nginx was trying to load. The team in question had never even checked the server.

It’s one thing to say ‘I don’t know what’s going on here and can you help me”. It’s a completely different thing to say “it’s a network problem, you need to fix it”, especially when it becomes abundantly clear that the team/person making this statement has done zero troubleshooting or due diligence and doesn’t even understand the details of the problem.

Occasionally we run into a genuine problem, yes they do occur. Unfortunately I’ve heard too many crying wolf stories that I almost never trust what I hear until I verify all the details myself.

How about this, I’ll take your credit card (Visa, Mastercard, Discover, no Amex) and if it’s a network problem I won’t bill you. However, if it’s not a network problem I’ll be sure to make it painful enough that the next time you’ll definitely do your homework before you come calling.

I’ll close this post out with the following line,

It’s Not Always the Networks Fault!

Cheers!

Oracle spams IT professionals blames network issue

by

Why is it that people still try to blame the network when either they don’t completely understand what broke or when they don’t want to admit what actually happened? It’s the digital equivalent of sweeping it under the rug… it was the networks fault.

Here’s a great example from Oracle. On Tuesday December 8th I received 12 messages from Oracle with the subject, “2016 Commerce Survey and a $25 Amazon Gift Card” between 6:00AM and 6:45AM EST. As a side note, they weren’t offering a $25 Amazon gift card for completing the survey, you would be entered to win a $25 Amazon gift card for completing the survey – see the difference?

This morning I received a follow-up from Zach Hanlon with the following explanation;

OracleNetworkIssue

If I’m an industry expert then please treat me as one and don’t just “blame the network”. I find it fascinating that there’s no mention of human error anywhere in the above statement. It’s certainly possible that “network problems” could have interrupted some process but I usually find that issues like these have multiple causes.

While I’m on my rant if you really value my input then you won’t insult my intelligence with that clickbait subject line next time.

Cheers!

War story from the frontlines of E-Commerce

by

I’m here to report that I survived my first holiday season working in retail. Thankfully my team and I were able to keep the network infrastructure humming along without any majors hiccups or issues which left everyone extremely happy including our customers. Here’s a short look behind the scenes of a our first big sale of the holiday leading up to Black Friday 2014.

It had all the markings of the Y2K war room, if you were around for that exciting event. There were no fewer than 30 people packed inside a large conference room which held 1 wall mounted 80′ TV, a floor stand mounted 65′ TV and a 50′ projector all cycling through a dizzying array of system, application, network performance metrics and sales statistics. There wasn’t a free seat to be had and there were a few people just standing in the background. There were developers, QA testers, marketing specialists, systems engineers, network engineers, mid-level managers (myself included) and a number of executives. There were no fewer than 6 power strips laid out across the conference room table with almost every outlet being spoken for. I’m not sure what Apple was thinking when they designed that power brick for the Mac books. At one end of the room, the 50′ projector was displaying the new Javascript/AJAX based sales dashboard against the wall. At the other end of the room was the 80′ wall mounted TV and floor stand mounted 65′ TV were both cycling through multiple system, application and network dashboards. It was curious to watch the business folks all watching the 50′ projector with the sales figures while the technical folks were all staring just as intently at the 80′ and 65′ TVs. I later noticed that there were another 20-25 people outside the conference room all staring just as intently at their laptop displays.

The sale was scheduled to go from 7:00PM to 11:00PM and had become the kick-off event to the holiday sales season for one of our brands. I had previously learned that there were significant technical issues in previous years so there was a considerable amount of stress and pressure to make sure that everything went off without a hitch. If there were any issues I was there to help quickly identify them and implement any needed workarounds to minimize any disruptions. There was no shortage of wise cracks regarding possible network, firewall or load-balancer issues as the event grew closer. In the months leading up to tonight we had run into a number of performance problems and technical hurdles so while I was confident in my team’s preparation and in the network infrastructure in general I knew success wasn’t necessarily assured. The application that was driving the website was overly resource, memory and bandwidth hungry which can be extremely problematic when trying to scale an application. While the website was utilizing a premiere Content Distribution Network (CDN) it wasn’t as optimized as some of our other brands and would occasionally pull 1MB+ HTML files from origin. While gains had been made in optimizing the website there were still concerns that the load on the web servers, application servers and database servers might be too much for the hardware to sustain.

Earlier that morning around 10:30AM an email campaign had gone out announcing the sale and that led to a significant jump in the number of sessions to the website along with a sizable increase in bandwidth utilization. A second email campaign would go out around 3:00PM which would again lead to an additional increase in the number of active sessions. At 6:30PM a push notification was sent to all mobile app (Apple and Android) users which caused yet another significant spike in the number of connections. At 7:00PM sharp the sale started with an impressive but steady stream of users shopping the site and checking out. There wasn’t much change until I noticed another increase in the number of connections to the load balancer around 8:30PM. After inquiring with the marketing team we learned there had been another email campaign sent to registered users. There were brief stints here and there were one or two application servers would slow down a little but nothing really significant until 10:30PM. Around that time there was yet another scheduled push notification made to all the mobile app users at which time more than half of the application servers came under a significant load each fielding more than 100 active connections from the load balancer. After a few minutes one of the application servers just couldn’t keep up and it became unresponsive, eventually failing out of the server farm with those customers that had been connected to that server being directed to one of the remaining application servers. A few minutes later the server recovered by itself and was automatically placed back into the server farm without any manual intervention and would run fine the rest of the night. When 11PM came we learned that the executive team decided to extend the sale an additional hour to 12AM midnight but the number of sessions and the traffic was already starting to wind down. We had made it through the 4 hour sale relatively unscathed. There was a small interruption to a few customers when one of the application servers went unresponsive but that issue only lasted about 5 minutes. It was a multi-million dollar event and I felt pretty good regarding the performance of the network infrastructure.

I had a few observations that night… email marketing campaigns if not properly throttled can inadvertently overload a website. Mobile push notifications can have the same affect although I would say they are more detrimental. While we saw significant click-thru conversions on the email marketing campaigns which in turn drove system and network utilization, the peaks out of mobile push notifications were much more severe. I’m guessing the conversion rate was much higher on the mobile app than the email campaigns and as such led to significant upticks in both system and network utilization. In short people were much more inclined to open a push notifications and immediately start shopping the website.

Now the work begins for next year’s holiday season. Here are a few of the bigger network infrastructure projects that myself and my team will be working on;

  • Upgrade Cisco 6509 pair from standalone to VSS cluster
  • Upgrade Cisco ACE 4710s to A10 Thunder 3030 Load Balancers
  • Design and deploy Aruba ClearPass for Network Access Control
  • Design and deploy Infoblox IPAM for DNS/DHCP Services with DNS Firewall

Cheers!

Image Credit: Sias van Schalkwyk

Network is Slow!

by

Those are the words that almost every network engineer loathes to hear, the “network is slow”. And those words are usually spoken by folks who really have no idea of how the network works let alone the understanding to quantify the word “slow”. In my past life I had built a fairly large dark fiber metropolitan network where the smallest link, outside of remote VPN offices, was 1Gbps. I spent years training the IT staff around me to understand the difference between a under performing application, an overloaded server and a congested (slow) network. In that past life I rarely heard the words, the “network is slow”. Fast forward to current day and it seems I have my work cutout for me since I hear those glowing words on average 3-4 times a day.

Add in 100Mbps to the desktop (thanks to the Avaya 4621 IP Phones) and you’ll even find me occasionally grinding my mouse in frustration as I wait for a large Visio diagram or Excel spreadsheet to open.  That’s not even mentioning the added delay if the document has been archived by an electronic vaulting or archiving solution. It’s even more grinding when that solution prompts me to authenticate in order to retrieve the archived document – that’s a great user experience.

Any reasonable user would blame the network because that’s the piece that ties everything together so that’s probably what’s broken. I wouldn’t fault the user for that assumption because they really don’t know any better, in their view it’s just plain slow and occasionally painfully so. So next time you hear the “network is slow”, just remember to smile!

Cheers!

Recent Posts

Categories

SUBSCRIBE BY EMAIL