host relay.verizon.net[206.46.232.11] refused
    to talk to me: 571 Email from 162.243.40.10 is currently blocked by Verizon
    Online's anti-spam system. The email sender or Email Service Provider may
    visit http://www.verizon.net/whitelist and request removal of the block.
    151214

Ok, it’s quite possible that someone on the discussion forums has been a misbehaving so I need to make sure there’s no legitimacy to this report. I need to scour the log files and make sure that there’s nothing going on. I keep the log files on my server for 30 days, let’s do some quick crude command line fu;

[root@moon ~]# grep -h @verizon /var/log/maillog* | awk '{ print $7 }' | sort | uniq -c
    427 to=nobody@verizon.net,

Note: I’ve obfuscated the email above so I don’t end up getting any more spam than I already receive daily.

Now that’s very interesting, I’m the only person that the server has been trying to mail which is getting denied by Verizon. Ok, so this problem is only impacting me, I guess that’s good.

So if it’s been going on for 30 days then I need to make sure the server is not listed on some RBL (real-time black list) somewhere. I’ll check http://www.blacklistalert.org/;

Alright so I wasn’t listed on any of the RBL, I looked through the logs for any other anomalies and found none, focusing again on mail for Verizon customers (verizon.net/verizon.com) and found nothing, I searched the discussion forums user database and blog comment subscriptions and found nothing. It must be a false positive on Verizon’s side, I’ll submit a request to Verizon following their instructions. I went to http://www.verizon.net/whitelist and I tried submitting a request as a Verizon customer and the form submission crashed with the following;

Ok, so I went back and submitted a request as an ISP (although I’m not an ISP but I’m starting to feel like one). That form was successfully submitted and I quickly received a reply via email.

That’s a boiler plate reply if I’ve ever seen one. Ok, so this doesn’t look like it’s going to be easy… I’ll need to chase the folks at abuse@verizon.net and probably in Twitter as well.

Cheers!

In short I’m just flat out impressed, it’s amazing how far technology has come and what you can do with a $35 device. I haven’t touched a breadboard in more than 20 years now but I’m excited to pick one up and start tinkering again. The electrical engineer in me is giddy with excitement and anticipation. While the computer scientist in me is eager to write some code and see how this little device can literally integrate into the real world.

I was impressed by how responsive the GUI was on the Raspberry Pi 2. It felt faster then some of the older Intel Core 3 laptops I have lying around the house. Using the USB wireless adapter that was included in the kit I purchased I was quickly able to get the Raspberry Pi to join my WPA2-PSK wireless network. Within 15 minutes I had a X windows desktop (you need to manually start X Windows with ‘startx’ from the command line interface – that reminded me of my early Linux days). I was also able to remotely connect via SSH using PuTTY to the little computer. The Raspberry Pi 2 uses a microSD card as the primary storage filesystem. I was again surprised by the performance of the 8GB Kingston microSD card that was included in the kit.  I’ve run a few live Linux distributions from CD/DVD and/or USB flash drives and the performance is always painfully slow. In this case the performance was transparent as the solution just worked and I didn’t need to bother about the bottlenecks because there were no visible performance issues.

Here are a few commands I used to update the Raspbian Linux distribution to the latest and greatest;

sudo apt-get update
sudo apt-get upgrade
sudo apt-get dist-upgrade

I’m going to document how to setup a Raspberry Pi 2 as a remote PRTG probe. That should be fun.

Are you using a Raspberry Pi or Arduino? What are you doing with it?

Cheers!

I'm troubleshooting a @Motorola WS5100 v3.3.5.0-002R that is being plagued with frequent high CPU utilization and generating 25,000+ pps

— Michael McNamara (@mfMcNamara) December 18, 2014

In the process of trying to understand the problem and come up with a solution I wanted to have better visibility and alerting when the problem actually occurred, I didn’t want to incur the delay that would involve the users calling the help desk and the help desk calling me. Thankfully there is a SYSLOG message recorded when an Access Port experiences a watchdog reset so I had a log message now I needed to find a way to alert on that message.

That’s where I turned to swatch, a handy little utility that will monitor log files for regular expressions and then take whatever action, such as ringing the console or sending an email message is configured. I installed swatch with relative ease thanks to yum and then set out to configure it appropriately.

I created the following configuration file;

#/etc/swatchrc

# swatchrc - define regular expressions and generate alerts when matches are found in logs
#            daemon is started from /etc/cron.d/swatch

# Motorola AP300 - malfunctioning AP ignore events from this device

ignore /00-A0-F8-ZZ-ZZ-ZZ/

# Motorola WS5100 Access Port Adoption Errors Reboot/Watchdog events

#Dec 20 07:53:07 ACME-WLS1 %CC-6-APREADOPTREASON: AP 00-A0-F8-XX-XX-XX readoption reason: ColdBoot/Watchdog
#Dec 20 07:53:25 ACME-WLS1 %CC-6-APREADOPTREASON: AP 00-A0-F8-XX-XX-XX readoption reason: Link failed

# Let's look for the phrase readoption and we'll alert of that text

watchfor /readoption/
        exec "echo '$_' | mail swatch -s 'SWATCH: Motorola WS5100 Adoption Issue' "
        threshold track_by=$6,type=limit,count=1,seconds=60
        echo=red
        bell 5

#end

In the swatch configuration I used the mail aliase of swatch so I edited the /etc/newaliases file to make sure that the entire team would receive the alert;

#
#  Aliases in this file will NOT be expanded in the header from
#  Mail, but WILL be visible over networks or from /bin/mail.
#
#       >>>>>>>>>>      The program "newaliases" must be run after
#       >> NOTE >>      this file is updated for any changes to
#       >>>>>>>>>>      show through to sendmail.
#

# Basic system aliases -- these MUST be present.
mailer-daemon:  postmaster
postmaster:     root

# General redirections for pseudo accounts.
bin:            root
daemon:         root
adm:            root
...
...
...
swatch:         root,mike,john,dan,tom

If the problem is extremely important I’ll usually add the the email SMS text message gateway for my provider. This way I’ll get both an email message and an SMS text message alerting me to the problem.

# Verizon SMS Text Messaging 123456789@vtext.com
# AT&T SMS Text Messaging 123456789@txt.att.net
# T-Mobile SMS Text Messsaging 123456789@tmomail.net
# Sprint SMS Text Messaging 123456789@messaging.sprintpcs.com

I made sure to recompile the aliases file with the newaliases command and then I set off to run swatch in the foreground of my SSH session.

[root@centos /]# swatch -c /etc/swatchrc -p 'tail -f -n 0 /var/log/loc1fac17.log'

*** swatch version 3.2.3 (pid:30643) started at Sat Dec 20 15:54:14 EST 2014

And I waited for the event.

Now I could go about doing some research and due diligence without worrying that I might inadvertently fail to spot the problem.

I’ll let you know how it turned out!

Cheers!

Note: This is a series of posts made under the Network Engineer in Retail 30 Days of Peak, this is post number 26 of 30. All the posts can be viewed from the 30in30 tag.

In this specific case all the routing was being performed by a number of high-end Cisco ASA firewalls which didn’t have PIM routing configured or enabled so I took the easy approach of just disabling IGMP snooping across the Cisco Catalyst 6504 and 4948 switches and the problem was solved. The cleaner solution would have been to setup an Mutlicast Router (mrouter) on the VLAN to properly handle all the IGMP requests and reports.

As pointed out by a colleague you can use a great little Python script written by RedHat for testing Multicast on your Linux servers.

Cheers!

Note: This is a series of posts made under the Network Engineer in Retail 30 Days of Peak, this is post number 3 of 30. All the posts can be viewed from the 30in30 tag.

Reference;
http://www.cisco.com/c/en/us/support/docs/switches/catalyst-6500-series-switches/68131-cat-multicast-prob.html
http://www.cisco.com/c/en/us/td/docs/security/asa/asa82/configuration/guide/config/route_multicast.html

In the early days of my foray into blogging I utilized Google’s Blogger for the first six months. I then decided to move to GoDaddy’s (shared) managed hosting which wasn’t as bad as some reviews would have you believe. A year later I decided to leave GoDaddy for RIMU Hosting. I left behind managed hosting for an un-managed CentOS Linux VPS (Virtual Private Server). While I was a former IBM AIX System Administrator and Linux enthusiast I wasn’t quite prepared for the effort required to setup and manage a simple Linux web server. While I enjoyed the challenge it took me quite sometime to get everything automated. As the traffic to my blog and the discussion forums grew I started running into the memory and bandwidth limitations of the plan I was using from RIMU so I decided to switch to Linode after reading some positive review (such as this one). I’m happy to say I’ve been using Linode for almost 18 months, since October 2011, and have never had any issues or problems.

There have been a number of significant upgrades at Linode over the past few months, so much so that I thought I would take a second to detail them here and shamelessly plug my referral link at the same time.

Linode NexGen: RAM Upgrade

Linode literally left the best for last since most virtual workloads are memory constrained. They are essentially bumping everyone up one level, a Linode 512 becomes a Linode 1G, a Linode 1024 becomes a Linode 2GB, so on and so forth. It should be noted that they are also increasing their pricing by $0.05 per month, example the Linode 1G is $20.00 and not $19.95.

Linode NextGen: The Hardware

Linode has upgraded their hosts with two Intel Sandy Bridge E5-2670 processors. The E5-2670 is at the high end of the power-price-performance ratio and each E5-2670 enjoys 20 MB of cache and has 8 cores running at 2.6 GHz. There’s a lot of processing power behind that virtual server depending on your needs.

Linode Nextgen: The Network

Linode has deployed a new Cisco Nexus 7000 and 5000 topology (very similar to the topology that I personally use) in their data centers. “To top things off we’ve increased the amount of outbound transfer included with all plans by 1,000%.  That’s right, 10 times the included transfer!”

Linode 512 upgraded from 200GB to 2000GB (2TB)
Linode 1G upgraded from 400GB to 4000GB (4TB)
Linode 2G upgraded from 800GB to 8000GB (8TB)
Linode 4G upgraded from 1600GB to 16000GB (16TB)
Linode 8G upgraded from 2000GB to 20000GB (20TB)

Storage increased by 20%

Linode 512 goes from 20GB to 24GB
Linode 1GB goes from 40GB to 48GB
Linode 2GB goes from 80GB to 96GB
Linode 4GB goes from 160GB to 192GB
Linode 8GB goes from 320GB to 384GB
Linode 12GB goes from 480GB to 576GB
Linode 20GB goes from 800GB to 960GB

My Thoughts

There’s definitely been quite a few changes over at Linode so I wondered what those changes might have done to performance. At the surface it certainly appears that the average Linode customer is now getting more. We’re getting 100% more memory, 20% more storage, 1,000% more bandwidth. So how what percent of a performance increase can we expect in processing?

We’ll I decided to run some UnixBench tests and compare these new tests with some previous results I posted in an article entitled Linode VPS Hosting I posted back in October 2011.

I started writing this article back in April 2013. That was before the Linode Manager password reset, which was explained by the security breach that was disclosed shortly thereafter. Throughout that time I’ve struggle to get performance numbers anywhere near what I captured in October 2011. I even attempted to engage Linode support and while they were cordial they gave me the typical ‘we can move you to a new host’ response without really engaging in an in-depth discussion around the horrendous performance numbers. I would write four or five paragraphs to which they would respond with one or two liners.

October 2011 Hardware

May 2013 Hardware

Here are the performance numbers of each side by side;

You can find the actual HTML results file online for October 2011 and May 2013.

It’s obvious that quite a few things have changed since I first tested Linode back in October 2011. The original testing was performed on a Linode 512 with 4 Intel Xeon L5520 @ 2.27GHz (4522.0 bogomips). The most recent testing was performed on a Linode 2048 with 8 Intel Xeon CPU E5-2630L 0 @ 2.00GHz (4000.1 bogomips). While the original hardware configuration offered 4 cores the latest hardware offering provides 8 cores. I’ve been using the 1 parallel process testing numbers to help gauge the performance of a single core. The disk IO numbers look very poor but when I perform a basic disk IO test everything seems pretty good.

[root@earth ~]# dd if=/dev/zero of=test bs=64k count=48k conv=fdatasync
49152+0 records in
49152+0 records out
3221225472 bytes (3.2 GB) copied, 45.4488 s, 70.9 MB/s

Perhaps the original data I collected in October 2011 was flawed, perhaps I was the only user on that physical server and now years later the Linode environment has become much more crowded – similar to Comcast Cable Modem Internet. It worked great the first few years but after everyone in the neighborhood started subscribing the performance really tanked.

I could probably use tools like Bonnie++ and Nbench to help validate my results but I wouldn’t be able to compare them against any previous results. I’d probably only use these tools if I was going to find a new hosting provider and wanted to benchmark their environments against what I have available today. I even went as far as to download UnixBench v5.1.3 and re-ran my tests only to score a 149.2 compared to the original result of 191.6.

With all that said the server and websites appear to be running fine.  The Web Page Performance tests for this site are pretty decent, 2.944 seconds (first view) and 1.623 seconds (second view). Perhaps the performance numbers will change when my server gets migrated to a host with the new Intel Xeon E5-2670 CPUs.

In summary I’m not sure what to say… I had thought this article would be an easy post to write but the performance numbers followed by the security incident have left me wondering if Linode is the hosting provider for me. Performance benchmarking within a virtual environment is really difficult given all the different components and the ever changing workloads.

Cheers!

I’m going to post a link here to the file, not quite sure why it was pulled from RedHat’s site.

virtio-win-1.1.16.vfd (MD5SUM: 7437f5d81fc43e8da3be01802fa4e9fb)

Cheers!

Every now and then a genuine (system administration) issue or problem surfaces that deserves some time and effort. Since I’m utilizing a virtual private server (VPS) running CentOS 5.5, I’m responsible for administering and managing the server myself. I was an IBM AIX (long live SMIT) and Solaris System Administrator in a previous life so it’s not a big challenge but it can be a time consuming task. The benefits of managing my own server are still significant enough for me and I’ve learned so much about Linux, MySQL, PHP, Perl, etc. that the experience has been well worth the investment in my view.

I recently noticed that I was getting a lot of bounced email messages on the server from a number of readers that had subscribed to posts on my blog. Here’s a quick snippet of the bounced error message;

Action: failed
Status: 5.1.7
Remote-MTA: dns; mx.acme.org
Diagnostic-Code: smtp; 550 5.1.7 ... We
    systematically reject 'apache@...'

It seems that a few domains (example above is acme.org – changed to protect identity) were rejecting any email message with the Return-Path set to apache@hostname. In my case the Return-Path was set to apache@michaelfmcnamara.com although the From address was set to noreply@michaelfmcnamara.com. Unfortunately you can’t set (not to my knowledge anyway) the Return-Path from within WordPress administration portal. You need to manually edit wp-includes/class-phpmailer.php and set the variable $Sender to the same email address you setup within WordPress to use as your From address.

/**
* Sets the Sender email (Return-Path) of the message.  If not empty,
* will be sent via -f to sendmail or as 'MAIL FROM' in smtp mode.
* @var string
*/
var $Sender            = 'noreply@michaelfmcnamara.com';

With that change complete I can see from the server logs (/var/log/maillog) that the Return-Path is now being properly set.

Feb 12 08:29:56 michaelfmcnamara postfix/pickup[9770]: 2B8FD2C3BB: uid=48 from=<noreply@michaelfmcnamara.com>
Feb 12 08:29:56 michaelfmcnamara postfix/cleanup[11068]: 2B8FD2C3BB: message-id=<67fa95dc7fd22d7c6cfd481d506bfd87@blog.michaelfmcnamara.com>
Feb 12 08:29:56 michaelfmcnamara postfix/qmgr[2647]: 2B8FD2C3BB: from=<noreply@michaelfmcnamara.com>, size=1729, nrcpt=1 (queue active)
Feb 12 08:29:56 michaelfmcnamara postfix/local[11070]: 2B8FD2C3BB: to=<whowhatwhen@michaelfmcnamara.com>, relay=local, delay=0.07, delays=0.04/0.01/0/0.02, dsn=2.0.0, status=sent (forwarded as 321C72C37A)
Feb 12 08:29:56 michaelfmcnamara postfix/qmgr[2647]: 2B8FD2C3BB: removed

With that change those domains that were rejecting email from my server are now accepting them again. Just another day where I’ve learned something new.

Cheers!

Update: Thursday February 24, 2011

It seems the upgrade to WordPress 3.1 has overwritten the change I made in the file… had to update the file again!

Update: Friday April 22, 2011

It seems the upgrade to WordPress 3.1.1 has overwritten the change I made in the file again!

I had installed Ubuntu using the original Ubuntu 8.04 LTS Desktop Edition which required 242 patches/upgrades to be installed after I installed the operating system. I believe Ubuntu has since released a slipstreamed version (8.04.1) with the latest and greatest patches and upgrades since June 2008. The initial install along with the subsequent updates was very painless, I just sat back and let the software do the work.

I was very pleased with the performance and the easy of use of Ubuntu. Now I just needed to find some solution to help make sure that my daughter didn’t mistakenly end up on some shady website. I stumbled across Glubble, a Firefox Add-on which allows you to control which websites your children can visit. Your child can also request access to additional websites which a parent must then authorize.

If you are a parent looking to provide a safe experience for your child while he/she surfs the Internet I would highly recommend Glubble!

Cheers!

On Tuesday July 10, 2008 a number of vendors including Microsoft, Cisco, Juniper and RedHat released patches and/or acknowledged the flaw existed. The Internet Software Consortium, the group responsible for development of the popular Berkeley Internet Domain Named (BIND) server from which nearly all DNS offshoots are based, also acknowledged the flaw and released a patch.

I personally spent about 90 minutes on last Wednesday updating several internal and external systems including numerous CentOS v5.2 servers and Windows 2003 Service Pack 2 servers. I was unable to find any mention of the DNS flaw on the Alcatel-Lucent website so I’ll probably need to place a call concerning Alcaltel-Lucent’s VitalQIP product.

I used yum to patch the CentOS Linux servers [“yum update”] and then just restarted the named process [“service named restart”]. On the Windows 2003 Service Pack 2 servers I used Windows Update to download and install KB941672 after which I rebooted the servers.

Here are some references:

http://www.theregister.co.uk/2008/07/09/dns_fix_alliance/
http://www.networkworld.com/news/2008/071008-patch-domain-name-servers-now.html
http://www.networkworld.com/news/2008/070808-dns-flaw-disrupts-internet.html

http://www.networkworld.com/podcasts/newsmaker/2008/071108nmw-dns.html

http://www.us-cert.gov/cas/techalerts/TA08-190B.html
http://www.microsoft.com/technet/security/bulletin/MS07-062.mspx

I would strongly suggest that all network administrators start looking into patching their DNS servers as soon as possible.

Cheers!

UPDATE: July 14, 2008

Here’s an update from RedHat concerning the configuration (named.conf) of BIND;

We have updated the Enterprise Linux 5 packages in this advisory. The default and sample caching-nameserver configuration files have been updated so that they do not specify a fixed query-source port. Administrators wishing to take advantage of randomized UDP source ports should check their configuration file to ensure they have not specified fixed query-source ports.

It seems that a check of the configuration file would be in order. Let me throw in a quick warning though if your DNS server is sitting behind a firewall you may need to check with the firewall administrator to understand how the firewall will behave if you randomize your source ports. I believe there are quite a few firewalls out there that only expect to see DNS traffic sourced from a DNS server on UDP/53.

Good Luck!

Just visit the current Mirrors list to start downloading today.

Note: Just be warned that if your running CentOS v5.0 or v5.1 you will be upgraded to CentOS v5.2 if you issue a “yum update“. I believe the release notes indicate you need to issue a “yum upgrade” in order to upgrade but that wasn’t my experience.

Cheers!