Linode Upgrades – Which hosting provider do you use?

Linode_LogoThere are dozens if not hundreds of hosting providers out there these days so how do you go about choosing the right one for you?

In the early days of my foray into blogging I utilized Google’s Blogger for the first six months. I then decided to move to GoDaddy’s (shared) managed hosting which wasn’t as bad as some reviews would have you believe. A year later I decided to leave GoDaddy for RIMU Hosting. I left behind managed hosting for an un-managed CentOS Linux VPS (Virtual Private Server). While I was a former IBM AIX System Administrator and Linux enthusiast I wasn’t quite prepared for the effort required to setup and manage a simple Linux web server. While I enjoyed the challenge it took me quite sometime to get everything automated. As the traffic to my blog and the discussion forums grew I started running into the memory and bandwidth limitations of the plan I was using from RIMU so I decided to switch to Linode after reading some positive review (such as this one). I’m happy to say I’ve been using Linode for almost 18 months, since October 2011, and have never had any issues or problems.

There have been a number of significant upgrades at Linode over the past few months, so much so that I thought I would take a second to detail them here and shamelessly plug my referral link at the same time.

Linode NexGen: RAM Upgrade

Linode literally left the best for last since most virtual workloads are memory constrained. They are essentially bumping everyone up one level, a Linode 512 becomes a Linode 1G, a Linode 1024 becomes a Linode 2GB, so on and so forth. It should be noted that they are also increasing their pricing by $0.05 per month, example the Linode 1G is $20.00 and not $19.95.

Plan RAM Disk XFER CPU Price
Linode 1G 1 GB 24 GB 2 TB 8 cores (1x priority) $20 / mo
Linode 2G 2 GB 48 GB 4 TB 8 cores (2x priority) $40 / mo
Linode 4G 4 GB 96 GB 8 TB 8 cores (4x priority) $80 / mo
Linode 8G 8 GB 192 GB 16 TB 8 cores (8x priority) $160 / mo
Linode 16G 16 GB 384 GB 20 TB 8 cores (16x priority) $320 / mo
Linode 24G 24 GB 576 GB 20 TB 8 cores (24x priority) $480 / mo
Linode 32G 32 GB 768 GB 20 TB 8 cores (32x priority) $640 / mo
Linode 40G 40 GB 960 GB 20 TB 8 cores (40x priority) $800 / mo

Linode NextGen: The Hardware

Linode has upgraded their hosts with two Intel Sandy Bridge E5-2670 processors. The E5-2670 is at the high end of the power-price-performance ratio and each E5-2670 enjoys 20 MB of cache and has 8 cores running at 2.6 GHz. There’s a lot of processing power behind that virtual server depending on your needs.

Linode Nextgen: The Network

Linode has deployed a new Cisco Nexus 7000 and 5000 topology (very similar to the topology that I personally use) in their data centers. “To top things off we’ve increased the amount of outbound transfer included with all plans by 1,000%.  That’s right, 10 times the included transfer!”

Linode 512 upgraded from 200GB to 2000GB (2TB)
Linode 1G upgraded from 400GB to 4000GB (4TB)
Linode 2G upgraded from 800GB to 8000GB (8TB)
Linode 4G upgraded from 1600GB to 16000GB (16TB)
Linode 8G upgraded from 2000GB to 20000GB (20TB)

Storage increased by 20%

Linode 512 goes from 20GB to 24GB
Linode 1GB goes from 40GB to 48GB
Linode 2GB goes from 80GB to 96GB
Linode 4GB goes from 160GB to 192GB
Linode 8GB goes from 320GB to 384GB
Linode 12GB goes from 480GB to 576GB
Linode 20GB goes from 800GB to 960GB

My Thoughts

There’s definitely been quite a few changes over at Linode so I wondered what those changes might have done to performance. At the surface it certainly appears that the average Linode customer is now getting more. We’re getting 100% more memory, 20% more storage, 1,000% more bandwidth. So how what percent of a performance increase can we expect in processing?

We’ll I decided to run some UnixBench tests and compare these new tests with some previous results I posted in an article entitled Linode VPS Hosting I posted back in October 2011.

I started writing this article back in April 2013. That was before the Linode Manager password reset, which was explained by the security breach that was disclosed shortly thereafter. Throughout that time I’ve struggle to get performance numbers anywhere near what I captured in October 2011. I even attempted to engage Linode support and while they were cordial they gave me the typical ‘we can move you to a new host’ response without really engaging in an in-depth discussion around the horrendous performance numbers. I would write four or five paragraphs to which they would respond with one or two liners.

October 2011 Hardware

System: li366-32: GNU/Linux
OS: GNU/Linux — 3.0.4-linode38 — #1 SMP Thu Sep 22 14:59:08 EDT 2011
Machine: i686: i386
Language: en_US.utf8 (charmap=”UTF-8″, collate=”UTF-8″)
CPUs: 0: Intel(R) Xeon(R) CPU L5520 @ 2.27GHz (4522.0 bogomips)
Hyper-Threading, MMX, Physical Address Ext
1: Intel(R) Xeon(R) CPU L5520 @ 2.27GHz (4522.0 bogomips)
Hyper-Threading, MMX, Physical Address Ext
2: Intel(R) Xeon(R) CPU L5520 @ 2.27GHz (4522.0 bogomips)
Hyper-Threading, MMX, Physical Address Ext
3: Intel(R) Xeon(R) CPU L5520 @ 2.27GHz (4522.0 bogomips)
Hyper-Threading, MMX, Physical Address Ext
Uptime: 11:06:54 up 14 min, 1 user, load average: 0.05, 0.04, 0.05; runlevel 3

May 2013 Hardware

System: earth.michaelfmcnamara.com: GNU/Linux
OS: GNU/Linux — 3.8.4-linode50 — #1 SMP Mon Mar 25 15:50:29 EDT 2013
Machine: i686: i386
Language: en_US.utf8 (charmap=”UTF-8″, collate=”UTF-8″)
CPUs: 0: Intel(R) Xeon(R) CPU E5-2630L 0 @ 2.00GHz (4000.1 bogomips)
Hyper-Threading, MMX, Physical Address Ext
1: Intel(R) Xeon(R) CPU E5-2630L 0 @ 2.00GHz (4000.1 bogomips)
Hyper-Threading, MMX, Physical Address Ext
2: Intel(R) Xeon(R) CPU E5-2630L 0 @ 2.00GHz (4000.1 bogomips)
Hyper-Threading, MMX, Physical Address Ext
3: Intel(R) Xeon(R) CPU E5-2630L 0 @ 2.00GHz (4000.1 bogomips)
Hyper-Threading, MMX, Physical Address Ext
4: Intel(R) Xeon(R) CPU E5-2630L 0 @ 2.00GHz (4000.1 bogomips)
Hyper-Threading, MMX, Physical Address Ext
5: Intel(R) Xeon(R) CPU E5-2630L 0 @ 2.00GHz (4000.1 bogomips)
Hyper-Threading, MMX, Physical Address Ext
6: Intel(R) Xeon(R) CPU E5-2630L 0 @ 2.00GHz (4000.1 bogomips)
Hyper-Threading, MMX, Physical Address Ext
7: Intel(R) Xeon(R) CPU E5-2630L 0 @ 2.00GHz (4000.1 bogomips)
Hyper-Threading, MMX, Physical Address Ext
Uptime: 21:42:19 up 9 days, 14:07, 1 user, load average: 0.18, 0.10, 0.06; runlevel 3

Here are the performance numbers of each side by side;

Test Score Unit Time Iters. Baseline Oct 2011 May 2013
Dhrystone 2 using register variables 16345243.3 lps 10.0 s 7 116700.0 1400.6 827.5
Double-Precision Whetstone 2455.5 MWIPS 10.1 s 7 55.0 446.5 301.2
Execl Throughput 1179.1 lps 30.0 s 2 43.0 274.2 184.7
File Copy 1024 bufsize 2000 maxblocks 342283.0 KBps 30.0 s 2 3960.0 864.4 139.6
File Copy 256 bufsize 500 maxblocks 87956.8 KBps 30.0 s 2 1655.0 531.5 91.9
File Copy 4096 bufsize 8000 maxblocks 958654.2 KBps 30.0 s 2 5800.0 1652.9 341.2
Pipe Throughput 488607.7 lps 10.0 s 7 12440.0 392.8 58.3
Pipe-based Context Switching 32606.8 lps 10.0 s 7 4000.0 81.5 23.1
Process Creation 2233.1 lps 30.0 s 2 126.0 177.2 108.3
Shell Scripts (1 concurrent) 2560.1 lpm 60.0 s 2 42.4 603.8 402.4
Shell Scripts (8 concurrent) 970.0 lpm 60.0 s 2 6.0 1616.7 1115.7
System Call Overhead 451501.4 lps 10.0 s 7 15000.0 301.0 185.4
System Benchmarks Index Score: 495.1 191.6

You can find the actual HTML results file online for October 2011 and May 2013.

It’s obvious that quite a few things have changed since I first tested Linode back in October 2011. The original testing was performed on a Linode 512 with 4 Intel Xeon L5520 @ 2.27GHz (4522.0 bogomips). The most recent testing was performed on a Linode 2048 with 8 Intel Xeon CPU E5-2630L 0 @ 2.00GHz (4000.1 bogomips). While the original hardware configuration offered 4 cores the latest hardware offering provides 8 cores. I’ve been using the 1 parallel process testing numbers to help gauge the performance of a single core. The disk IO numbers look very poor but when I perform a basic disk IO test everything seems pretty good.

[root@earth ~]# dd if=/dev/zero of=test bs=64k count=48k conv=fdatasync
49152+0 records in
49152+0 records out
3221225472 bytes (3.2 GB) copied, 45.4488 s, 70.9 MB/s

Perhaps the original data I collected in October 2011 was flawed, perhaps I was the only user on that physical server and now years later the Linode environment has become much more crowded – similar to Comcast Cable Modem Internet. It worked great the first few years but after everyone in the neighborhood started subscribing the performance really tanked.

I could probably use tools like Bonnie++ and Nbench to help validate my results but I wouldn’t be able to compare them against any previous results. I’d probably only use these tools if I was going to find a new hosting provider and wanted to benchmark their environments against what I have available today. I even went as far as to download UnixBench v5.1.3 and re-ran my tests only to score a 149.2 compared to the original result of 191.6.

With all that said the server and websites appear to be running fine.  The Web Page Performance tests for this site are pretty decent, 2.944 seconds (first view) and 1.623 seconds (second view). Perhaps the performance numbers will change when my server gets migrated to a host with the new Intel Xeon E5-2670 CPUs.

In summary I’m not sure what to say… I had thought this article would be an easy post to write but the performance numbers followed by the security incident have left me wondering if Linode is the hosting provider for me. Performance benchmarking within a virtual environment is really difficult given all the different components and the ever changing workloads.

Cheers!

Avaya Contact Center Agent Desktop Display Quietly Crashing

869352_72885464I thought I would share this story… it’s another story of “it’s the network’s fault” when in reality it really has nothing to-do with the network but it falls to the network engineers and consultants to prove the point beyond a reasonable doubt.

I can’t tell you how it irks me to hear people say “it’s the networks fault” when they have absolutely no clue as to how anything works and have no data to support their wild claims. I would think a lot more of them if they just said, “I’m sorry, I haven’t got a frigging clue what’s happening here but can you help me?” And of course the problem always needs to be resolved yesterday as if the building itself was on fire.

We have multiple Avaya Aura Contact Center (formerly Nortel Symposium) installations. At one of these locations we began receiving trouble tickets that the Agent Desktop Display (ADD) which is a small software application that listens to a Multicast stream and displays a ticker tape banner showing the contact center queue details was quietly closing after only a few minutes of running on the local desktop/laptop. The local telecom technician verified that the problem only occurred on a specific floor, the users on the other floors had no issues or problems. A quick check of the core Avaya Ethernet Routing Switch 8600 and edge Avaya Ethernet Routing Switch 5520s indicated that IGMP and PIM were configured and working properly.

Note:A few years back now I detailed how to configure IGMP, DVMRP and PIM for Multicast routing.

I asked the local telecom technician to perform a packet trace so I could see what was happening on the wire. The packet trace indicated that the desktop/laptop was issuing an IGMP leave request and was closing the HTTP/TCP socket it had open to the web server so that was proof enough for me that the application was silently crashing and the operating system was cleaning up all the open ports and IGMP sessions.

6376 2013-05-16 07:47:46.052281 10.1.46.144 10.1.38.55 TCP     54   3317 > 80 [RST, ACK] Seq=4467 Ack=10430 Win=0 Len=0
6377 2013-05-16 07:47:46.052595 10.1.46.144 224.0.0.2  IGMPv2  46   Leave Group 230.0.0.2

The actual Multicast stream from the application/web server was fine;

6353	2013-05-16 07:47:43.995183	10.1.38.55	230.0.0.2	UDP	511	Source port: 1031  Destination port: 7040
6354	2013-05-16 07:47:43.995502	10.1.38.55	230.0.0.2	UDP	502	Source port: 1025  Destination port: 7050
6355	2013-05-16 07:47:43.995885	10.1.38.55	230.0.0.2	UDP	813	Source port: 1026  Destination port: 7030
6356	2013-05-16 07:47:43.996301	10.1.38.55	230.0.0.2	UDP	860	Source port: 1032  Destination port: 7020
6357	2013-05-16 07:47:43.996505	10.1.38.55	230.0.0.2	UDP	343	Source port: 1033  Destination port: 7060
6358	2013-05-16 07:47:43.996726	10.1.38.55	230.0.0.2	UDP	331	Source port: 1027  Destination port: 7070
6359	2013-05-16 07:47:43.996886	10.1.38.55	230.0.0.2	UDP	153	Source port: 1028  Destination port: 7110
6360	2013-05-16 07:47:43.997048	10.1.38.55	230.0.0.2	UDP	153	Source port: 1034  Destination port: 7100
6361	2013-05-16 07:47:43.997199	10.1.38.55	230.0.0.2	UDP	135	Source port: 1030  Destination port: 7090
6362	2013-05-16 07:47:43.997371	10.1.38.55	230.0.0.2	UDP	135	Source port: 1036  Destination port: 7080
6363	2013-05-16 07:47:43.997525	10.1.38.55	230.0.0.2	UDP	127	Source port: 1035  Destination port: 7120
6364	2013-05-16 07:47:43.997647	10.1.38.55	230.0.0.2	UDP	127	Source port: 1029  Destination port: 7130

The packet trace did show some odd UDP broadcast traffic from one specific desktop that happen to be running GE’s Centricity Perinatal (CPN). This is a software application used to monitor Labor & Delivery, the Nursery and the NICU. We use it to actually monitor, chart and graph the strips put out by the fetal monitors. There’s a software component of the GE CPN solution called B-Relay which is the piece of software that floods the VLAN with all those UDP broadcasts. Unfortunately this UDP flooding is by design and is required for the application to function properly.

6205	2013-05-16 07:47:26.710685	10.1.47.210	10.1.47.255	UDP	251	Source port: 1759  Destination port: 7005
6206	2013-05-16 07:47:26.853810	10.1.47.210	10.1.47.255	UDP	822	Source port: 1760  Destination port: 7043
6211	2013-05-16 07:47:28.215486	10.1.47.210	10.1.47.255	UDP	60	Source port: 1783  Destination port: 7013

Looking at the packet traces I quickly noticed that while there are multiple destination ports they are overlapping between 7001 and 7999. I would theorize that the GE CPN software was eventually hitting a UDP port that the ADD software was listening on and since it was a broadcast packet it tried to process the data and was quietly choking and crashing. I shutdown the Ethernet port connecting the GE CPN desktop and had the local telecom technician run his test again. He called back about 30 minutes later to let me know that everything was working fine and that whatever I had done had fixed the problem. Well it wasn’t really fixed because now I had to figure out how to get both applications to co-exist.

The solution was to isolate the GE CPN desktops to their own VLAN so that the UDP broadcasts wouldn’t hit the closet VLAN where the Contact Center users resided. Another possible solution might have been to try and change the UDP ports that either GE CPN or Avaya ADD software was using but that change would have probably taken weeks if not months. I was able to spin up a new VLAN in about 30 minutes and get everyone back up and running again.

Have you got a story to share? I’d love to hear it!

Cheers!

Home Desktop Upgrade 2013

antec_p280It’s been a while since I upgraded my primary desktop machine so I thought it was time to spend a little coin. While I did install a SSD back in August 2012 I thought it was time for a completely new machine. When I get a new machine I usually pass my old one down to the wife and then her old machine to the kids and so on this way everyone has a little something to be excited about. This time around I decided to splurge and go for what I would probably consider a fairly high-end build. I also ordered a new case and power supply so I could have both machines running at the same time and could take my time migrating the data and content from the old machine to the new machine.

Here’s what I purchased from NewEgg;

I thought about going with water cooling but eventually decided to stick with air cooling and purchased the Zalman CNPS9900ALED .The Zalman heatsink is insanely huge, words just don’t do it justice. While the Antec P280 case is fairly large itself, the Zalman heatsink quickly fills the case making it look small. Thankfully everything arrived within a few days of ordering it and it took me about 2 days to assemble the new machine. I performed the assembly in the following order;

  1. Install Corsair power supply into Antec P280 case
  2. Install G.Skill memory onto Gigabyte motherboard
  3. Install Intel Core i7 CPU onto Gigabyte motherboard
  4. Install Zalman heatsink onto Intel Core i7 CPU and Gigabyte motherboard
  5. Install Gigabyte motherboard into case
  6. Install Samsung 840 SSD into case
  7. Install Western Digital hard drive into case
  8. Install MSI 6950 video card into Gigabyte motherboard and case
  9. Wire everything together, including all power leads, front panel LEDs, USB headers, SATA connectors, etc.

The hardest part was figuring out how to mount the Zalman CPU heatsink which probably took about 20-35 minutes. Thankfully everything was working right out of the box and I had no DoAs or RMAs to contend with. Obviously I stayed with Windows 7 64-bit having heard all about the issues with Windows 8.

That leaves me with the following build;

  • Windows 7 Service Pack 1 64-bit
  • Intel Core i7-3370K @3.5Ghz
  • 16GB G.Skill Ripjaws DDR3 2133 memory (4 sticks)
  • Gigabyte GA-Z77X-UD5H motherboard
  • MSI R6950 Twin Frozr II OC Radeon HD 6950 2GB
  • 256GB Samsung 840 Pro Series SSD
  • 2TB Western Digital Black SATA 3 hard disk
  • LITE-ON Blue Ray DVD burner
  • Antec P280 case
  • CORSAIR HX Series HX850 power supply
  • Logitech G15 keyboard
  • Logitech G5 mouse
  • Logitech C910 webcam
  • 2 x ASUS 27″ LCD display
  • Canon i2600 InkJet Printer

Cheers!