Michael McNamara

technology, networking, virtualization and IP telephony

Black Friday – the aftermath

by

I had been hoping for a relatively quiet day yesterday but it was anything of the sort. We had been on a conference bridge around 11:30PM on Thursday night to monitor the system for a number of marketing campaigns that were kicking off at 12AM Friday morning. While there was a significant increase in the number of online shoppers (sessions) by 1AM Friday morning everything was running smoothly. We reconvened in the office on Friday morning around 7AM and for the first 2 hours things were relatively quiet. We could see a steady increase in the number of sessions and bandwidth being served from origin but for the most part everything looked pretty good. It wasn’t until the mobile app push came that the boat sprang a few leaks.

Oracle Website DownThen came the news that Best Buy was down which only served to excite the room even more. We run the same back-end E-Commerce solution as Best Buy. Which in a twist of irony that vendor’s website is currently down as I write this post on Saturday morning.

We had started failing behind such that we had a combined 180,000+ TCP connections on our Tier1 and Tier2 firewalls (we would usually have around 21,000 TCP connections). The load-balancers were also getting a workout as the number of open TCP connections started climbing first from the usual double digits into five digits.  While the websites were slowing down they were still working but the number of open TCP connections possibly indicated that we were starting to fall behind and corrective action would be necessary or we would eventually fall over and crash.

The assembled team dug into the data using tools from New Relic, Zabbix and Splunk to name a few and quickly found issues with API calls from the mobile app that were in need of tuning. The team worked diligently to try and steady the ship by spinning up additional back-end instances and by adding additional threads/workers to our existing caching layers, web and app tiers. Then we discovered something odd, we started observing packet loss on traffic egressing our Tier1 perimeter firewall. That certainly hadn’t been there before but it was there now. Was it a result of the extreme number of sessions that were hitting the websites or had something else broken. The team decided to spin up additional data centers and started shedding new customers to other data centers which helped keep the existing data centers from falling over. We weren’t sure if the problem with the firewall would get worse so spinning up additional data centers allowed us some flexibility if things went south. With the other data centers online and running we decided to fail over the HA firewall pair to see if that would correct the packet loss issue and sure enough the problem disappeared after we failed over.

I never thought we would have seen such a basic issue as packet loss within the internal network but we did. We were able to quickly identify the issue and we took corrective action to remedy the problem. As for the root cause, I’m not sure we’ll ever know with 100% certainty but we’ll probably be implementing some yearly preventative maintenance as that firewall had been up some 680 days.

After a few hours we spun down the extra data centers that we had brought online now that we had addressed and resolved the issues we found. While there were a few traffic spikes throughout the remaining afternoon and evening everything ran smoothly. In the end the executive team was happy with the outcome and result and I was relieved that I had survived Black Friday with only a few bruises. Thankfully we didn’t have any issues on the retail store side or with our call centers or distribution centers. It was the E-Commerce that had literally consumed my entire day.

I’m sitting here writing this post contemplating what’s in store for Cyber Monday. Which for me will actually kick off around 8PM Sunday night. I’m told that in past years we’ve generally seen a 20-30% increase in traffic between Black Friday and Cyber Monday.

Cheers!

Note: This is a series of posts made under the Network Engineer in Retail 30 Days of Peak, this is post number 6 of 30. All the posts can be viewed from the 30in30 tag.

Black Friday – Is is storming yet?

by

What’s the Internet weather look like today?

While it was snowing in the Philadelphia, PA suburbs on Wednesday – I do love a white Thanksgiving, today I’m going to be focused entirely on the Internet weather. I’ll most likely be spending the majority of my day in the war room carefully watching all the different metrics and assuring board members, executives and directors that everything is working smoothly. While I can control the technical pieces to the puzzle I have little control over the actual sales  numbers although it’s not a stretch for people to blame technology when sales numbers fail to meet projections.

What are some resources that I use to monitor the overall health of the Internet outside of my data centers?

Cheers!

Note: This is a series of posts made under the Network Engineer in Retail 30 Days of Peak, this is post number 5 of 30. All the posts can be viewed from the 30in30 tag.

 

 

Happy Thanksgiving!

by

Happy Thanksgiving!

This year the Thanksgiving holiday has special significance given that I’m now working in the retail industry. Thanksgiving is quite literally the calm before the storm. The storm that is Black Friday and Cyber Monday. Those two days are the most important shopping days out of the entire calendar year to almost every retailer. What’s my biggest concern? That Black Friday and Cyber Monday will be too good, so much so that various front-end and back-end systems will fail and/or collapse under the surge of orders and shopping users. I’m not talking about just web based E-Commerce but also the traditional brick and mortar store fronts and all the POS (Point of Sale) transactions and credit charges that are sure to flood our data centers, servers and networks. Black Friday also marks the start of the holiday shopping season which runs through Christmas. That’s a total of 28 days where my team and I will need to be on our game and make sure we quickly address any issues or problems that might arise. Any downtime during the holiday peak is very painful to the bottom line. If for instance you have a website that takes in $1,000,000 an hour in sales and you’re down for even just 10 minutes, that equates to ~ $166,000 in lost sales. If we’re down for 30 minutes that’s $500,000 in potential lost sales. In this case downtime can literally cost you your job and then some. If that wasn’t enough the recent cyber break-ins at some of the largest retailers has everyone on edge.

It’s my belief that we’ve adequately prepared ourselves and that the infrastructure and architecture we’ve assembled including the Internet Service Providers and Content Delivery Networks will meet the surging demand over the next 28 days.

I definitely hope to enjoy the turkey, stuffing, carrots, mashed potatoes, sweet potatoes, corn and gravy this year, not to mention the ice cream and apple pie. Here in the states, football or American Football as it’s referred to abroad is a very popular sport. I also hope to take in some of the football game between my local Philadelphia Eagles and the Dallas Cowboys. The wife cheerfully reminded me after proof-reading this post that I’ll also be spending some time in the kitchen doing the dishes! ;)

Cheers!

Note: This is a series of posts made under the Network Engineer in Retail 30 Days of Peak, this is post number 4 of 30. All the posts can be viewed from the 30in30 tag.

Network Engineer in Retail – 30 Days of Peak

by

I’m a big fan of Greg Ferro and so I’m happy to accept his challenge of posting 30 blogs in 30 days although I’m going to try and put my own spin on it. Having been in the healthcare industry for almost 17 years and education for 3 years before that this is my first year in the retail industry and it’s been pretty exciting.

In healthcare I had 50,000+ ports under my stead and was charged with the care of critical life-safety networks. In retail I have under my care ~ 14,000+ ports and the responsibility of maintaining connectivity for multiple e-commerce websites that can generate in excess of $1,000,000 an hour in sales.

While there certainly are differences between education, healthcare and the retail industries they are all still looking for the same thing albeit with differing levels of redundancy and (high) availability.

I can’t promise 30 brilliant posts, not even sure I can deliver one, but I do promise a musing look into 30 days of a married father of three (girls), working as a manager for a larger retailer responsible for voice, data and storage networking during the holiday peak (November 25 and December 25).

It should be if nothing else slightly entertaining but please know that you’ve been warned, the next 30 days or so could be slightly entertaining or highly aggravating.

Cheers!

PS: I’m happily accepting guests posts from whomever is interested, I might even have the family through out one or two.

Recent Posts

Categories

SUBSCRIBE BY EMAIL