Michael McNamara

technology, networking, virtualization and IP telephony

Retail Holiday Peak 2016

by

It’s that time of year again… the holidays are just around the corner and every retailer is gearing up for Black Friday and Cyber Monday. My employer kicked off the holiday shopping season last night with one brand having their yearly 4-hour sale. Thankfully there were no surprises and our infrastructure and application stack was able to handle the additional load without issue. I did stumble upon an instrumentation issue between PRTG and a Cisco FirePOWER 4110 firewall – perhaps I’ll share more about that problem in another post. It’s a challenge every year to try and forecast the potential load and then meet the surge in demand, let’s not forget about all the email marketing campaigns and app push notifications that the brands want to hit their customers with. It can be a very challenging time for many Information Technology teams.

Now we wait for Thanksgiving and the four days to follow… confident that we’ve taken all the correct steps and everything is ready.  Only time will tell the true story.

Cheers!

How much security is enough?

by

We had a lively round table debate about “how much security is enough?” during Networking Field Day 11. It’s certainly not a pure networking question which some in the room debated is no longer, or perhaps has never been, the network engineer’s responsibility, but a large number of networking professionals these days are still charged with keeping the digital landscape clear of threats within their employers networks.

The argument put forth was essentially that it is cheaper for companies to take the data breach hit than feed the ever growing IT security budgets because there are no penalties or little downsides for the many business that are involved in what has become a daily occurrence of customer and/or credit card data theft from a resulting data breach. Greg suggests that companies might be better suited investing in a good public relations firm to help manage any public crisis that might arise. I wouldn’t agree that there aren’t any downsides although I would reluctantly agree that large businesses appear to be emerging relatively unscathed from these incidents. The emergence of data breach insurance, also known as cyber liability insurance, gives additional credence that large business look at security and data breaches as a simple math problems.

The formula might look like this;

( (Revenue Loss + Breach Related Costs) – Breach Insurance) < IT Security Spend

In short the financial penalty for losing your customer data doesn’t justify the IT security spend needed to actually sure the the data. So it’s cheaper for large businesses to essentially take the financial hit for a data breach rather than spend the considerable resources need to secure the data, application or solution.

There’s certainly validity to the overall point that there’s little motivation for large businesses to spend significant resources on overall IT security. In an article entitled, “Why companies have little incentive to invest in cybersecurity” by Benjamin Dean, Benjamin provides numerous facts and supporting evidence to suggest that there’s little motivation for large businesses to heavily invest in protecting customer information. Benjamin provides data from both the Target and Home Depot breaches that supports the argument and ultimately ponders if additional governmental oversight will be needed to close the loop.

I would counter with this point, when has any large business spent any more than it absolutely needed on anything.  I can’t tell you how often I’ve stood in front of a budget committee and been told that I’ll just need to make do with the capital or operating funds that I have available no matter the strategic importance to the business operation or ROI.

What do you think?

Is additional government oversight needed to get large business to take more responsibility?

Cheers!

Image Credit: Aap Deluxe

Black Friday 2015, behind the scenes

by

I was reading a post from Paul Stewart entitled, “Black Friday, Technology Glitches and Revenue Lost“, last night and thought to myself – there’s a blog post in there somewhere.

As someone who works for a large global retailer I have a new found appreciation for the challenges and issues that present themselves during the holiday season for retailers. It’s been a very tiring week for myself and a large number of my co-workers as we worked tirelessly from Thursday night (Thanksgiving) to Tuesday morning to keep any issues like what Paul’s wife experienced from impacting our brands. In the past two years I’ve learned that scale brings a whole other dimension to the game. I’m not just talking about bandwidth, that’s usually a pretty simple problem to solve. Instead I’m referring to all the inter-dependencies in both the application layer and in the hardware (networking, storage, compute). Let me try and put this into perspective, over the long weekend one of the brands we manage attracted over 7.9 million visitors to their website generating some 234 million page views. In comparison this blog gets about 1,500 page views a day just to put that number into perspective.

Why is it so hard?

For these few weeks of the year most sites will see significantly more traffic than they see throughout the year. Discount sales will also drive additional volume to the sites which may have issues trying to keep up with the number of online users. Often time the issue is scale… can the application scale… can the infrastructure scale to meet the demand. And more importantly can the vendors and third-parties that you rely on also scale to meet the demand. Look at the issues Target experienced on Cyber Monday when they offered 15% off every item. Around 10AM that morning they started having load issues and had to turn on Akamai’s Shopper Prioritization Application (SPA) which essentially holds users in a queue within Akamai’s network only allowing a fixed number of users through the door to keep the website from completely collapsing under the load.

In my role I’m generally concerned with the following infrastructure;

  • Internet Service Providers (availability of the websites from multiple peering points)
  • Internet Bandwidth (bandwidth and performance to/from the websites from a global viewpoint)
  • Internet Load Balancers (balancing the external load across multiple external facing web servers)
  • Internet Firewalls (protecting origin servers)
  • Internal Bandwidth (performance within and between data centers)
  • Internal Load Balancers (load balancing internal API services)
  • Internal Firewalls (filtering traffic between web/app/db tiers)
  • Storage Fabric Performance (bandwidth utilization on individual SAN switches)
  • Storage Array Performance (read/write latency per storage pool or LUN)

When we have problems we’ll often see connection counts on our external and internal firewalls spike as the web or app servers try to spin up additional processes and connections to compensate either for the load or for the previously failed query or connection. This is often just a symptom of the problem and not the cause of the problem. The cause might be a third-party API that’s not responding quickly enough and because that API is backing up, now our application is backing up because we’re relying on them. In another example a poorly written stored procedure severely impacted a database, while some folks believed the database was at fault, it wasn’t hard to quickly identify the poorly written SQL code as the culprit. And while this stored procedure worked for 100 users it quickly failed when there were 1,000 users on the website.

Here’s one that people often forget about…

  • Voice Infrastructure – Contact Center 1-800 PRI/T1/SIP channel availability

We have 10 T1 circuits into one of our Call Centers, providing some 240 channels (ability to handle 240 concurrent calls). Throughout the year we rarely ever get above 60 concurrent calls. On Black Friday and Cyber Monday though we ran at 100% utilization (240 calls in queue) for quite a few hours each day, keeping our agents extremely busy.

Isn’t the cloud all about scalability?

Yes, it certainly is but not every retailer is as big as Amazon and has the ability to re-architect their entire application stack around the cloud. We purposely spin up additional web and app server instances just prior to the holiday season and we spend a lot of time running load tests to validate that everything is working properly.  Not every retailer has either the resources or the staff to bulk up for the holiday rush.

The majority of large retailers definitely track cart abandonment, some retailers will even remind you of an item left in your cart and occasionally entice you with an additional discount or coupon code. Go add something to your cart at NewEgg.com and see how long it is before they drop you an email message.

Retailers rely on tools like Pingdom, AlertSite, New Relic, AppDynamics and AppBoy to provide visibility into their website or mobile application performance along with the user experiences (user timings).

So while it might seem like a pretty easy problem to quickly solve, it’s actually a very complex problem.

Cheers!

Image Credit: Philippe Ramakers

War story from the frontlines of E-Commerce

by

I’m here to report that I survived my first holiday season working in retail. Thankfully my team and I were able to keep the network infrastructure humming along without any majors hiccups or issues which left everyone extremely happy including our customers. Here’s a short look behind the scenes of a our first big sale of the holiday leading up to Black Friday 2014.

It had all the markings of the Y2K war room, if you were around for that exciting event. There were no fewer than 30 people packed inside a large conference room which held 1 wall mounted 80′ TV, a floor stand mounted 65′ TV and a 50′ projector all cycling through a dizzying array of system, application, network performance metrics and sales statistics. There wasn’t a free seat to be had and there were a few people just standing in the background. There were developers, QA testers, marketing specialists, systems engineers, network engineers, mid-level managers (myself included) and a number of executives. There were no fewer than 6 power strips laid out across the conference room table with almost every outlet being spoken for. I’m not sure what Apple was thinking when they designed that power brick for the Mac books. At one end of the room, the 50′ projector was displaying the new Javascript/AJAX based sales dashboard against the wall. At the other end of the room was the 80′ wall mounted TV and floor stand mounted 65′ TV were both cycling through multiple system, application and network dashboards. It was curious to watch the business folks all watching the 50′ projector with the sales figures while the technical folks were all staring just as intently at the 80′ and 65′ TVs. I later noticed that there were another 20-25 people outside the conference room all staring just as intently at their laptop displays.

The sale was scheduled to go from 7:00PM to 11:00PM and had become the kick-off event to the holiday sales season for one of our brands. I had previously learned that there were significant technical issues in previous years so there was a considerable amount of stress and pressure to make sure that everything went off without a hitch. If there were any issues I was there to help quickly identify them and implement any needed workarounds to minimize any disruptions. There was no shortage of wise cracks regarding possible network, firewall or load-balancer issues as the event grew closer. In the months leading up to tonight we had run into a number of performance problems and technical hurdles so while I was confident in my team’s preparation and in the network infrastructure in general I knew success wasn’t necessarily assured. The application that was driving the website was overly resource, memory and bandwidth hungry which can be extremely problematic when trying to scale an application. While the website was utilizing a premiere Content Distribution Network (CDN) it wasn’t as optimized as some of our other brands and would occasionally pull 1MB+ HTML files from origin. While gains had been made in optimizing the website there were still concerns that the load on the web servers, application servers and database servers might be too much for the hardware to sustain.

Earlier that morning around 10:30AM an email campaign had gone out announcing the sale and that led to a significant jump in the number of sessions to the website along with a sizable increase in bandwidth utilization. A second email campaign would go out around 3:00PM which would again lead to an additional increase in the number of active sessions. At 6:30PM a push notification was sent to all mobile app (Apple and Android) users which caused yet another significant spike in the number of connections. At 7:00PM sharp the sale started with an impressive but steady stream of users shopping the site and checking out. There wasn’t much change until I noticed another increase in the number of connections to the load balancer around 8:30PM. After inquiring with the marketing team we learned there had been another email campaign sent to registered users. There were brief stints here and there were one or two application servers would slow down a little but nothing really significant until 10:30PM. Around that time there was yet another scheduled push notification made to all the mobile app users at which time more than half of the application servers came under a significant load each fielding more than 100 active connections from the load balancer. After a few minutes one of the application servers just couldn’t keep up and it became unresponsive, eventually failing out of the server farm with those customers that had been connected to that server being directed to one of the remaining application servers. A few minutes later the server recovered by itself and was automatically placed back into the server farm without any manual intervention and would run fine the rest of the night. When 11PM came we learned that the executive team decided to extend the sale an additional hour to 12AM midnight but the number of sessions and the traffic was already starting to wind down. We had made it through the 4 hour sale relatively unscathed. There was a small interruption to a few customers when one of the application servers went unresponsive but that issue only lasted about 5 minutes. It was a multi-million dollar event and I felt pretty good regarding the performance of the network infrastructure.

I had a few observations that night… email marketing campaigns if not properly throttled can inadvertently overload a website. Mobile push notifications can have the same affect although I would say they are more detrimental. While we saw significant click-thru conversions on the email marketing campaigns which in turn drove system and network utilization, the peaks out of mobile push notifications were much more severe. I’m guessing the conversion rate was much higher on the mobile app than the email campaigns and as such led to significant upticks in both system and network utilization. In short people were much more inclined to open a push notifications and immediately start shopping the website.

Now the work begins for next year’s holiday season. Here are a few of the bigger network infrastructure projects that myself and my team will be working on;

  • Upgrade Cisco 6509 pair from standalone to VSS cluster
  • Upgrade Cisco ACE 4710s to A10 Thunder 3030 Load Balancers
  • Design and deploy Aruba ClearPass for Network Access Control
  • Design and deploy Infoblox IPAM for DNS/DHCP Services with DNS Firewall

Cheers!

Image Credit: Sias van Schalkwyk

Conclusion – 30 Days of Peak

by

It was back on November 24, 2014 that I accepted Greg’s challenge to write 30 blog posts in 30 days. I’ve made it to day 30 and I’m still here, although I’m honestly more than a little tired and burnt out. Working for a retail company makes the Christmas holidays extremely stressful – the holiday peak is extremely important for every retailers balance sheet so there’s extra scrutiny around any technical issues whether they are actual or just perceived. I’ve heard horror stories of past holiday peaks so it appears that this holiday has been pretty successful. We had no major issues in our stores or on our websites and thankfully we didn’t have any major issues on Black Friday or Cyber Monday. We had a few challenges in one of our distribution centers thanks to ICMPv6 traffic flooding the network. Overall I’d like to think that I had something to-do with that success, probably not at as much as I’d like to take credit for. I’ve got a great team working for me so I need to credit them as well. I had been at my previous employer for 17 years and had the network infrastructure pretty well greased and running without issues, I’ve come to the realization that it’s going to take me a lot longer than 12 months to get the network infrastructure where I’d like it to be with my new employer.

As I mentioned when I started this challenge that I thought there might only be a handful of decent posts.

There were a few posts that I wrote but after I thought about them for a while I decided not to post them. I now work for a publicly traded company so I need to be mindful of not inadvertently publishing something that isn’t public knowledge – such as sales figures. I also work for a global retailer that’s been the target of some very intricate hacking attempts so I need to be mindful of what I reveal about our security posture. I have some stories from consulting engagements that would make great posts, but again I need to be mindful of my clients privacy even if I don’t name them outright. Especially if they are repeat clients which most of mine are.

Here are my favorite posts from the past 30 days;

Cheers!

Image Credit: Ted C.

Note: This is a series of posts made under the Network Engineer in Retail 30 Days of Peak, this is post number 30 of 30. All the posts can be viewed from the 30in30 tag.

Recent Posts

Categories

SUBSCRIBE BY EMAIL