Michael McNamara

technology, networking, virtualization and IP telephony

Opinion – responsible reporting of shooting suspects

by

I’m going to stray from my normal focus and pen an opinion regarding the recent mass shootings in the United States and the mainstream news media coverage. You may or may not agree with me but the purpose of the post isn’t really intended to have you agree with me, it’s more for me to 1) express my opinion and 2) encourage others to share their opinion.

Why is it that major news media outlets continue to publish the name(s) and or pictures of these villains?

While there is no such thing as the “Mass Shooting Bill”, I would think that common sense would start to prevail after more than 300 recorded mass shootings in the United States already in 2015. Why do the major news media outlets continue to provide a platform to publicize these notorious and infamous villains? I fear the news media inspiring additional copycat killers just so they too can get their 5 minutes of fame. We don’t need to know their name, read their name, hear their name or see their picture. I’m all for the First Amendment here folks but when can we expect common sense and human decency to trump ratings and ad sales. How about exclusively covering the victims, and the so many other lives that are left in shambles.

Cheers!

Black Friday 2015, behind the scenes

by

I was reading a post from Paul Stewart entitled, “Black Friday, Technology Glitches and Revenue Lost“, last night and thought to myself – there’s a blog post in there somewhere.

As someone who works for a large global retailer I have a new found appreciation for the challenges and issues that present themselves during the holiday season for retailers. It’s been a very tiring week for myself and a large number of my co-workers as we worked tirelessly from Thursday night (Thanksgiving) to Tuesday morning to keep any issues like what Paul’s wife experienced from impacting our brands. In the past two years I’ve learned that scale brings a whole other dimension to the game. I’m not just talking about bandwidth, that’s usually a pretty simple problem to solve. Instead I’m referring to all the inter-dependencies in both the application layer and in the hardware (networking, storage, compute). Let me try and put this into perspective, over the long weekend one of the brands we manage attracted over 7.9 million visitors to their website generating some 234 million page views. In comparison this blog gets about 1,500 page views a day just to put that number into perspective.

Why is it so hard?

For these few weeks of the year most sites will see significantly more traffic than they see throughout the year. Discount sales will also drive additional volume to the sites which may have issues trying to keep up with the number of online users. Often time the issue is scale… can the application scale… can the infrastructure scale to meet the demand. And more importantly can the vendors and third-parties that you rely on also scale to meet the demand. Look at the issues Target experienced on Cyber Monday when they offered 15% off every item. Around 10AM that morning they started having load issues and had to turn on Akamai’s Shopper Prioritization Application (SPA) which essentially holds users in a queue within Akamai’s network only allowing a fixed number of users through the door to keep the website from completely collapsing under the load.

In my role I’m generally concerned with the following infrastructure;

  • Internet Service Providers (availability of the websites from multiple peering points)
  • Internet Bandwidth (bandwidth and performance to/from the websites from a global viewpoint)
  • Internet Load Balancers (balancing the external load across multiple external facing web servers)
  • Internet Firewalls (protecting origin servers)
  • Internal Bandwidth (performance within and between data centers)
  • Internal Load Balancers (load balancing internal API services)
  • Internal Firewalls (filtering traffic between web/app/db tiers)
  • Storage Fabric Performance (bandwidth utilization on individual SAN switches)
  • Storage Array Performance (read/write latency per storage pool or LUN)

When we have problems we’ll often see connection counts on our external and internal firewalls spike as the web or app servers try to spin up additional processes and connections to compensate either for the load or for the previously failed query or connection. This is often just a symptom of the problem and not the cause of the problem. The cause might be a third-party API that’s not responding quickly enough and because that API is backing up, now our application is backing up because we’re relying on them. In another example a poorly written stored procedure severely impacted a database, while some folks believed the database was at fault, it wasn’t hard to quickly identify the poorly written SQL code as the culprit. And while this stored procedure worked for 100 users it quickly failed when there were 1,000 users on the website.

Here’s one that people often forget about…

  • Voice Infrastructure – Contact Center 1-800 PRI/T1/SIP channel availability

We have 10 T1 circuits into one of our Call Centers, providing some 240 channels (ability to handle 240 concurrent calls). Throughout the year we rarely ever get above 60 concurrent calls. On Black Friday and Cyber Monday though we ran at 100% utilization (240 calls in queue) for quite a few hours each day, keeping our agents extremely busy.

Isn’t the cloud all about scalability?

Yes, it certainly is but not every retailer is as big as Amazon and has the ability to re-architect their entire application stack around the cloud. We purposely spin up additional web and app server instances just prior to the holiday season and we spend a lot of time running load tests to validate that everything is working properly.  Not every retailer has either the resources or the staff to bulk up for the holiday rush.

The majority of large retailers definitely track cart abandonment, some retailers will even remind you of an item left in your cart and occasionally entice you with an additional discount or coupon code. Go add something to your cart at NewEgg.com and see how long it is before they drop you an email message.

Retailers rely on tools like Pingdom, AlertSite, New Relic, AppDynamics and AppBoy to provide visibility into their website or mobile application performance along with the user experiences (user timings).

So while it might seem like a pretty easy problem to quickly solve, it’s actually a very complex problem.

Cheers!

Image Credit: Philippe Ramakers

Motorola Handheld Scanners – Cisco WLC – IBM AS400

by

It’s the network’s fault? Well no not really but here’s another entertaining story…

Within the past six months my employer opened a new 1 million square foot distribution facility here in southeastern Pennsylvania. It’s been an exciting challenge for me personally. One of the bigger challenges has been getting the wireless network tuned in since the building opened essentially empty and now almost 4 months later is filled to the gills with product as we prepare for the upcoming holiday season. It’s about 45′ to the ceiling framing joists so I had our integrator mount the APs (Cisco AP 2702e) to 6′ of electrical conduit hanging from the ceiling. This way the APs are a little closer to the intended coverage space yet they are high enough to avoid being hit by the many fork lifts and other vehicles that traverse the distribution center.

IMG_20150908_154833902Weekly I review the power settings and RF coverage as the building literally fills up with product…which attenuates the signal from the wireless APs little by little. The majority of my WiFi experience is in hospitals, very old buildings and corporate office space where the RF signal only covers about 40-50′ at max and sometimes much smaller (hospitals). The biggest surprise in this facility is the propagation of the RF signal, there’s literally too much RF signal. I’ve had to turn off quite a few AP radios especially in the 2.4Ghz (802.11b/g) band to keep the channel interference to a minimum.

While RF coverage issues are no surprise to anyone working in the industry I have had an interesting time tracking down an idle timeout issue between the Motorola handheld scanners and the IBM AS400 host which runs our warehouse inventory system. The default idle user timeout on the Cisco WLC 5508 is 5 minutes, so I had to changed that parameter to 43200 seconds (12 hours). I also changed the session timeout to 86400 seconds (24 hours). Our employee shifts run 10 hours so these settings looked like they would work – and they do work, except they didn’t work for everyone. In my initial testing I found that I could login from one side of the distribution center, walk to the other side of the distribution center (6 minute walk – it’s that big) and I could pickup my session where I left off without issue even with the handheld going to sleep as I physically moved through the distribution center. I thought the problem was solved and moved onto other issues only to eventually hear that some users were still reporting that they were being “kicked out” – but not everyone.

It was a very specific set of users… those working the receiving dock and the wholesale racking. What was the common denominator? These users would occasionally put down their Motorola handhelds for upwards of 10-15 minutes as they unloaded a truck or prepared to put away a pallet. When they tried to use the handheld it would prompt that their AS400 session had ended and they would need to log in again.

I ran a few tests from the office space in the distribution center and quickly determined that if I was idle longer than 12 minutes I would generally get disconnected from my AS400 session. Was this issue coming from the wireless network or the AS400 host itself? I setup a quick and easy control test. I enabled telnet on one of my Linux jump boxes and established a connection from my test MC9190. After logging into my Linux jumpbox I sat the device on my desk for 25 minutes with no input from me. The Motorola MC9190 went to sleep in about 30 seconds, 25 minutes later I picked up the device, waited for it to power up and connect to the network and found my telnet session still working without issue so this test essentially eliminated the wireless network.

It turns out that the AS400 uses a TCP keepalive to determine if a user is still connected to the system. I’m guessing that since the handheld is sleeping it’s not responding to the keepalive packets so the host system is eventually closing the connection to the handheld. I have some inquires into our AS400 team and IBM to validate my theory.

Cheers!

Update: Sunday October 25, 2015

We were able to update the session keep-alive parameter on the IBM AS400 host. This value defaults to 10 minutes. So every 10 minutes the host will check if the session is still valid by sending a keep-alive packet. If the IBM AS400 host doesn’t get a response to the check it will logoff the session. We increased this value to 20 minutes and that’s helped greatly.

Raspberry Pi 2 Model B – Let the fun begin!

by

I finally broke down and ordered a Raspberry Pi 2 Model B and here are my first thoughts after tinkering with this $35 device for a few hours. The default username is “pi” while the default password is “raspberry”.

In short I’m just flat out impressed, it’s amazing how far technology has come and what you can do with a $35 device. I haven’t touched a breadboard in more than 20 years now but I’m excited to pick one up and start tinkering again. The electrical engineer in me is giddy with excitement and anticipation. While the computer scientist in me is eager to write some code and see how this little device can literally integrate into the real world.

I was impressed by how responsive the GUI was on the Raspberry Pi 2. It felt faster then some of the older Intel Core 3 laptops I have lying around the house. Using the USB 2015-08-21-230509_1632x1018_scrotwireless adapter that was included in the kit I purchased I was quickly able to get the Raspberry Pi to join my WPA2-PSK wireless network. Within 15 minutes I had a X windows desktop (you need to manually start X Windows with ‘startx’ from the command line interface – that reminded me of my early Linux days). I was also able to remotely connect via SSH using PuTTY to the little computer. The Raspberry Pi 2 uses a microSD card as the primary storage filesystem. I was again surprised by the performance of the 8GB Kingston microSD card that was included in the kit.  I’ve run a few live Linux distributions from CD/DVD and/or USB flash drives and the performance is always painfully slow. In this case the performance was transparent as the solution just worked and I didn’t need to bother about the bottlenecks because there were no visible performance issues.

Here are a few commands I used to update the Raspbian Linux distribution to the latest and greatest;

sudo apt-get update
sudo apt-get upgrade
sudo apt-get dist-upgrade

I’m going to document how to setup a Raspberry Pi 2 as a remote PRTG probe. That should be fun.

Are you using a Raspberry Pi or Arduino? What are you doing with it?

Cheers!

ISC BIND 9.10.2-P3 Forwarding Caching Only Nameserver

by

I recently had to migrate a large DNS environment from about 23 Microsoft Domain Controllers to Infoblox DNS. I could have just deleted all the zones and set the forwarding on the Microsoft DNS servers but I wanted to leave the Microsoft DNS configuration and data in place to provide a quick backout option in the unlikely event that it was need (it was needed but the second time around using the named.conf file below was the charm).

PrintI ended up deploying ISC BIND 9.10.2-P3 across a mix of Windows 2003 and Windows 2008 domain controller servers, some 32-bit and some 64-bit.

As I alluded to above I originally had issues running BIND getting error messages such as the following after only a few hours running the service and clients failing to get name resolution.

27-Jul-2015 19:15:04.575 general: error: ..\client.c:2108: unexpected error:
27-Jul-2015 19:15:04.575 general: error: failed to get request's destination: failure
27-Jul-2015 19:15:04.981 general: error: ..\client.c:2108: unexpected error:
27-Jul-2015 19:15:04.981 general: error: failed to get request's destination: failure
27-Jul-2015 19:15:20.971 general: error: ..\client.c:2108: unexpected error:
27-Jul-2015 19:15:20.971 general: error: failed to get request's destination: failure

There were also a few other errors that apeared to be releated to the anti-DDoS mechanisms built into BIND;

27-Jul-2015 19:50:02.369 resolver: notice: clients-per-query increased to 15

So I went back and recrafted the named.conf file and came up with the following which seems to be working well for me now almost 5 days after the Infoblox DNS migration.

You’ll noticed that I commented out the localhost zone and the 127.0.0.1 reverse zone as well. I didn’t think that BIND would run without them but sure enough it does. I also enabled query logging so I could see what type of abuse the DNS servers were getting. I found a couple of servers that were querying more than 40,000 times a minute for a management platform that had been retired almost 5+ years ago.

options {
  directory "c:\program files\isc bind 9\bin";
 
  // here are the servers we'll send all our queries to
  forwarders {10.1.1.1; 10.2.2.2;};
  forward only;

  auth-nxdomain no;

  // need to include allow-query at a minimum
  allow-recursion { "any"; };
  allow-query { "any"; };
  allow-transfer { "none"; };

  // lets leave IPv6 off for now less to worry about
  listen-on-v6 { "none"; };

  // standard stuff
  version none;
  minimal-responses yes;
 
  // cache positive and negative results for only 5 minutes
  max-cache-ttl 300;
  max-ncache-ttl 300;

  // disable DDoS mechanisms in BIND
  clients-per-query 0;
  max-clients-per-query 0;

};

logging{
   channel example_log{
    file "C:\program files\isc bind 9\log\named.log" versions 3 size 250k;
    severity info;
    print-severity yes;
    print-time yes;
    print-category yes;
  };

  channel queries_file {
    file "c:\program files\isc bind 9\log\queries.log" versions 10 size 10m;
    severity dynamic;
    print-time yes;
  };

  category default{ example_log; };
  category queries { queries_file; };

};

//zone "localhost" in{
//  type master;
//  file "pri.localhost";
//  allow-update{none;};
//};

//zone "0.0.127.in-addr.arpa" in{
//  type master;
//  file "localhost.rev";
//  allow-update{none;};
//};

I setup my first nameserver running BIND 4.x back in 1995, more than 20 years ago while working at Manhattan College. While I'm pretty familiar with BIND a lot has changed since then and so I had to-do a fair bit of research to arrive at the configuration above.

Hopefully someone else will find it helpful.

Cheers!

Recent Posts

Categories

SUBSCRIBE BY EMAIL