Michael McNamara

technology, networking, virtualization and IP telephony

Internet Utilization at 99.9% Arrgghhh!

by

Man_on_phone_2I thought I would just share this short story with you all… it’s a classic case of what can happen even with the best of plans and intentions. We recently deployed Adobe Acrobat Reader 9.1.2 via Microsoft Active Directory Group Policy.

We rushed the deployment in order to address some of the recent Acrobat vulnerabilities that were being actively exploited in the wild by Nine-Ball and other trojans/malware. We noticed an unusual uptick in Internet utilization almost immediately after the package had been deployed. When we examined our Websense logs we found an extreme number of HTTP requests to swupd.adbobe.com. We determined that these requests were coming from Adobe software products that were attempting to check for an update via Adobe’s auto-update feature. The HTTP requests were being denied by our Blue Coat ProxySG appliances because we require user authentication to access the Internet. While the Adobe auto-update component was able to read the PAC file configured within Internet Explorer it was not able to provide authentication when challenged with a 407 response. We originally thought the sheer number of clients making requests was putting an undo burden on the system so we added some CPL code to our Blue Coat ProxySG appliances to allow non-authenticated access to *.adobe.com. Within minutes of that change the wheels on the bus came flying off literally.

We just happen to have two 50Mbps Ethernet links to the Internet being served from two Blue Coat ProxySG appliances with about 5,500 client PCs. Within minutes both ProxySG appliances went to 96% CPU utilization and both Internet links went to 99.9% utilization. We had literally let the cat out of the bag and it was off and running… the number of client PCs trying to download updates from Adobe surged and they literally started to choke our two Internet connections.

Thankfully the Blue Coat ProxySG appliances support bandwidth classes. We created a 1Mbps class and added some CPL code to bandwidth restrict access to *.adobe.com. While that proved to be the quick fix we’re also deploying an update via Group Policy to disable the auto-update feature per Adobe’s knowledgebase article.

Cheers!

Blue Coat ProxySG Appliances Load Balancing & High Availability

by

We just recently deployed four Blue Coat ProxySG 510 appliances along with two Blue Coat ProxyAV 810 appliances in our enterprise. The Blue Coat ProxySG appliance is essentially a proxy server that integrates with NTLM/AD authentication and content filtering. The Blue Coat ProxyAV appliance is essentially an Anti-Virus solution, you choose which AV engine to license from Kaspersky, Sophos, McAfee or Panda, for all Internet content.

pic-sg510 We have two Internet links located at different geographically locations. We placed two Blue Coat ProxySG 510 appliances at one location along with a single Blue Coat ProxyAV 810 appliance. We setup each ProxySG with it’s own VIP and made each box a backup to the other. We duplicated that same configuration at the second geographic location with two Blue Coat ProxySG 510s and a single Blue Coat ProxyAV 810. The folks that are familiar with the Blue Coat product line might think that the 510 model is probably somewhat undersized for the size of our environment but our design was to use all four 510s in an active/active configuration as opposed to an active/standby configuration. In the event of a appliance failure we may have some high utilization but we won’t suffer an outage.

How did we achieve the load balancing and high availability?

It’s not really that difficult, although let’s not tell the folks writing the checks that. You’ll need to use a PAC file with some special JavaScript code to help provide the poor mans load balancing. The VIPs will provide the redundancy within each geographic site and we’ll trust the browser to provide the redundancy between the two geographic sites.

//
// Blue Coat Proxy Auto-Configuration File (PAC)
//
// This PAC file will provide a poor mans load balancing solution by
// hasing the IP address of the client to help spread the load of the
// client across two different ProxySG appliances. The variables at
// the top of the PAC file need to be adjusted to the specific VIP
// addresses of each ProxySG appliance.
//
function FindProxyForURL(url, host)
{

    // Declare Variables
    var OLDPROXY = "PROXY 10.1.20.100";

    var HECTOR = "PROXY 10.1.127.60";
    var FERGAL = "PROXY 10.1.127.65";
    var MEG = "PROXY 10.2.127.60";
    var SEAN = "PROXY 10.2.127.65";

    var SITE1VIP1 = "PROXY 10.1.127.62";
    var SITE1VIP2 = "PROXY 10.1.127.63";
    var SITE2VIP1 = "PROXY 10.2.127.62";
    var SITE2VIP2 = "PROXY 10.2.127.63";

    var ipSubs = myIpAddress().split(".");
    var localIP = myIpAddress();

    ret = URLhash(url);

    // Use DIRECT if the destination is just a plain WINS name
    if (isPlainHostName (host) ||
       // Use DIRECT if the host is within the michaelfmcnamara.com domain space
       dnsDomainIs(host, ".michaelfmcnamara.com") ||
       // Use DIRECT if the host is within the RFC1918 ten-dot-zero address space
       isInNet(host, "10.0.0.0", "255.255.0.0") ||
    return "DIRECT"; 

    // Use DIRECT if the URL starts with 10.*
    if (shExpMatch(url, "http://10.*") ||
       shExpMatch(url, "https://10.*"))
    return "DIRECT";

   // Use the Blue Coat Proxy for an remaining URL matching "http:"
   if( url.substring(0, 5) == "http:" ) {
      if (isInNet(myIpAddress(), "10.1.0.0", "255.255.0.0") {
         if ( (ipSubs[3] % 2) == 0 ) {
            return SITE1VIP1 + ":8080";
         } else {
            return SITE1VIP2 + ":8080";
         }
      } else {
         if ( (ipSubs[3] % 2) == 0 ) {
            return SITE2VIP1 + ":8080";
         } else {
            return SITE2VIP2 + ":8080";
         }
      }
   } else if( url.substring(0, 6) == "https:" )
      if (isInNet(myIpAddress(), "10.1.0.0", "255.255.0.0") {
         if ( (ipSubs[3] % 2) == 0 ) {
            return SITE1VIP1 + ":8080";
         } else {
            return SITE1VIP2 + ":8080";
         }
      } else {
         if ( (ipSubs[3] % 2) == 0 ) {
            return SITE2VIP1 + ":8080";
         } else {
            return SITE2VIP2 + ":8080";
         }
      }
   } else if( url.substring(0, 5) == "ftp:" ) {
      if (isInNet(myIpAddress(), "10.1.0.0", "255.255.0.0") {
         if ( (ipSubs[3] % 2) == 0 ) {
            return SITE1VIP1 + ":8080";
         } else {
            return SITE1VIP2 + ":8080";
         }
      } else {
         if ( (ipSubs[3] % 2) == 0 ) {
            return SITE2VIP1 + ":8080";
         } else {
            return SITE2VIP2 + ":8080";
         }
      }
   } else if( url.substring(0, 5) == "mms:" ) {
      if (isInNet(myIpAddress(), "10.1.0.0", "255.255.0.0") {
         if ( (ipSubs[3] % 2) == 0 ) {
            return SITE1VIP1 + ":1775";
         } else {
            return SITE1VIP2 + ":1775";
         }
      } else {
         if ( (ipSubs[3] % 2) == 0 ) {
            return SITE2VIP1 + ":1775";
         } else {
            return SITE2VIP2 + ":1775";
         }
      }
   } else {
      return "DIRECT";
   }
}

// This line is just for testing; you can ignore it.  But, if you are having
// problems where you think this PAC file isn't being loaded, then change this
// to read "if (1)" and the alert box should appear when the browser loads this
// file.
//
// This works for IE4, IE5, IE5.5, IE6 and Netscape 2.x, 3.x, and 4.x.
// This does not work for Mozilla (and probably not for Netscape 6.x).
// (For IE6, tested on Win2K)
//
if (0) {
    alert("DEBUG: PAC file LOADED");
}

function URLhash(name)
{
var  cnt=0;
    var str=name.toLowerCase(name);
    if ( str.length ==0) {
         return cnt;
    }
    for(var i=0;i < str.length ; i++) {
       var ch= atoi(str.substring(i,i + 1));
        cnt = cnt + ch;
    }

    return cnt ;
}

/* 
   URLhash2( ) for directory name hash computing version.
   written by SHARP Corp in Feb 1997
   
   Objects in a same directory will be accessed via the same proxy.
   Use URLhash2( ) instead of URLhash( ) if you prefer to use persistent
   connection in HTTP 1.1

   http://www.sharp.co.jp/sample/test/img/mebius.png
   http://www.sharp.co.jp/sample/test/img/zaurus.png
   http://www.sharp.co.jp/sample/test/img/wiz.png
   <------------------------------------->
         directory name hashing here
   
*/

function URLhash2(name)
{
var  cnt=0;
var  dirptr=0;

    var str=name.toLowerCase(name);
    if ( str.length ==0) {
         return cnt;
    }

/* skip filename in directory */
    for(var i=str.length - 1;i >=0 ; i--) {
        if ( str.substring(i,i +1) == '/' ) {
            dirptr = i+1 ;
            break;
        }
    }

    for(var i=0;i < dirptr; i++) {
       var ch= atoi(str.substring(i,i + 1));
        cnt = cnt + ch;
    }

    return cnt ;
}

function atoi(charstring)
{

    if ( charstring == "a" ) return 0x61; if ( charstring == "b" ) return 0x62;
    if ( charstring == "c" ) return 0x63; if ( charstring == "d" ) return 0x64;
    if ( charstring == "e" ) return 0x65; if ( charstring == "f" ) return 0x66;
        if ( charstring == "g" ) return 0x67; if ( charstring == "h" ) return 0x68;
    if ( charstring == "i" ) return 0x69; if ( charstring == "j" ) return 0x6a;
    if ( charstring == "k" ) return 0x6b; if ( charstring == "l" ) return 0x6c;
    if ( charstring == "m" ) return 0x6d; if ( charstring == "n" ) return 0x6e;
    if ( charstring == "o" ) return 0x6f; if ( charstring == "p" ) return 0x70;
    if ( charstring == "q" ) return 0x71; if ( charstring == "r" ) return 0x72;
    if ( charstring == "s" ) return 0x73; if ( charstring == "t" ) return 0x74;
    if ( charstring == "u" ) return 0x75; if ( charstring == "v" ) return 0x76;
    if ( charstring == "w" ) return 0x77; if ( charstring == "x" ) return 0x78;
    if ( charstring == "y" ) return 0x79; if ( charstring == "z" ) return 0x7a;
    if ( charstring == "0" ) return 0x30; if ( charstring == "1" ) return 0x31;
    if ( charstring == "2" ) return 0x32; if ( charstring == "3" ) return 0x33;
    if ( charstring == "4" ) return 0x34; if ( charstring == "5" ) return 0x35;
    if ( charstring == "6" ) return 0x36; if ( charstring == "7" ) return 0x37;
    if ( charstring == "8" ) return 0x38; if ( charstring == "9" ) return 0x39;
    if ( charstring == "." ) return 0x2e;
    return 0x20;
}

Let me provide some explanation of the PAC file attached above. First you may notice that I have the function URLhash and URLhash2 included in the PAC file. I don’t actually use those functions although I may in the future.

There are two geographic sites (10.1.0.0/16 and 10.2.0.0/16) each with two Blue Coat ProxySG appliances and a single ProxyAV appliance. Overall there are four ProxySG appliances and two ProxyAV appliances. The PAC file will essentially route all requests “statically” to the local ProxySG appliances  with the script load balancing the client requests, depending on the client’s IP address, to one of the two local ProxySG appliances. If you wanted geographic redundancy you could alter the “return” statement like so;

"PROXY 10.1.127.62:8080; 10.2.127.62:8080"

In this case the browser will attempt to contact the proxy server at 10.1.127.62:8080 and if the browser is unable to contact that proxy it will then try the proxy server at 10.2.127.62:8080. You’ll need to make sure that you include the (semicolon) “;” between each proxy server.

This solution has been working really well. The Blue Coat ProxySG appliance has a really impressive policy and rule base. You can do almost anything you want including coaching pages and warning banners. We current have our Blue Coat ProxySG appliances running Websense (onbox) for content filtering.

Cheers!

Recent Posts

Categories

SUBSCRIBE BY EMAIL