Michael McNamara

Blue Coat ProxySG Health Checks

Michael McNamara — Wed, 17 Apr 2013 02:49:58 +0000

You always need to be keeping your “eyes” on everything… well have the Blue Coat ProxySG monitor itself so you can sit back and be more productive.

Here’s a quick example of how to create a ‘Health Check’ on a Blue Coat ProxySG appliance to help notify you when the appliance can no longer reach the Internet. In the example below I decided to poll Google and Yahoo each every minute. The actual health check looks for a “200 OK” similar to the help check of a Layer 7 switch or content load balancer.

!- BEGIN health_checks
health-check
default e-mail healthy enable
default e-mail sick enable
default event-log healthy information
default event-log sick severe
default snmp healthy enable
default snmp sick enable
create icmp "user.Internet" 10.1.1.10
edit "user.Internet"
interval healthy 60
interval sick 60
exit
create http "user.InternetTest1" http://www.google.com
edit "user.InternetTest1"
interval healthy 60
interval sick 120
exit
create http "user.InternetTest2" http://www.yahoo.com
edit "user.InternetTest2"
interval healthy 60
interval sick 120
exit
exit
!- END health_checks

Cheers!

Packet Capture on a Blue Coat ProxySG Appliance

Michael McNamara — Wed, 08 Dec 2010 06:13:58 +0000

A large number of colleagues recently had their leased laptops refreshed with a new model and they chose to install Windows 7, a decision which I can’t really fault. Unfortunately as with many things these days we quickly found a few issues, one of which was with Microsoft’s Live Messenger and our Blue Coat ProxySG appliances. I almost immediately suspected the Blue Coat ProxySG appliance so I decided to look there first.

I started with a quick packet capture on the Blue Coat ProxySG appliance. I logged into the appliance and setup a filter to capture just the traffic to/from the IP address of my laptop. Our proxy servers have thousands of clients utilizing them at any one time so trying to perform a packet capture without a filter would be equivalent to trying to take a drink of water from a fire hydrant.

[root@linux ~]# telnet proxy.acme.org
Trying 10.1.2.100...
Connected to proxy.acme.org (10.1.2.100).
Escape character is '^]'.

Using telnet exposes your password. Using ssh is a safer choice.

Username: admin
Password: ********
SG800>enable
Enable Password: ********
SG800#pcap filter expr "host 10.1.1.2"
ok
SG800#pcap start
ok

With the packet trace running I went about duplicating the problem from my laptop.

SG800#pcap stop
ok
SG800#pcap info
packet capture information:
Packets captured:               76
Bytes captured:               9034
Packets written:                76
Bytes written:               11290
Max packet ram:                  0
Packet ram used:                 0
Packets filtered:             6113
Bridge capture all:       Disabled
Current state:             Stopped
Filtering:                      On
Filter expression:                 expr "host 10.1.1.2"

SG800#pcap transfer ftp://ftp.acme.org username password

ok

I retrieved the packet capture from the FTP server and used WireShark to examine the data. The packet trace showed the proxy server sending a RST packet closing the connection to the SOCKS port.

Another trick I learned a long time ago with the ProxySG appliances; try https://proxy.acme.org:8082/SYSInfo (you should obviously replace the hostname in the preceding URL if you want to try it out on your ProxySG appliance). In the IM debug information I found the following reference;

6983.574 MSN 5523C010: Can't proceed with unknown version: MMSG-VER-MSNP18
6983.574 MSN 5523C010: VER: MMSG-VER-MSNP18 not supported
6983.560 Admin: Accepted SOCKS client connection from 10.1.1.2 (policySession: 3895168, socksTrans: 7B5A004)

The log entries above suggest that this latest version of the Live Messenger is using a protocol version that isn’t yet supported by the Blue Coat ProxySG appliance, or at least isn’t supported in the 4.3.x software branch. I did a quick Google search and found a relative knowledge-base article on Blue Coat’s website that provides all the details around which IM versions are specifically supported on which ProxySG software releases. There is an option to just tunnel the SOCKS connection which I’ll need to investigate.

Cheers!

Blue Coat ProxySG Appliance – test http url

Michael McNamara — Thu, 19 Aug 2010 02:00:14 +0000

We recently had an issue where a few of our Blue Coat ProxySG appliances were having issues connecting to a specific URL. We would continually get the “TCP Error” banner from the Blue Coat proxy servers trying to connect to this website. We had no issues connecting directly (outside of the ProxySG appliances) so the proxy servers were assumed to be the immediate suspect in the problem.

Thankfully there’s a way to quickly and easily test access a specific URL from the CLI interface of the Blue Coat ProxySG appliances.

SG800#test http get ?

So I performed a quick test from the CLI interface;

SG800#test http get http://someurlsomewhere.com

Type escape sequence to abort.
Executing HTTP get test

* HTTP request header sent:
GET http://someurlsomewhere.com HTTP/1.0
Host: someurlsomewhere.com
User-Agent: HTTP_TEST_CLIENT

* HTTP response header recv'd:
HTTP/1.1 503 Service Unavailable
Cache-Control: no-cache
Pragma: no-cache
Content-Type: text/html; charset=utf-8
Proxy-Connection: close
Connection: close
Content-Length: 1329

Measured throughput rate is 0.03 Kbytes/sec
HTTP get test passed

It was easy to immediately see that the web server was returning a 503 error to the proxy server, something that was impossible to see from the client browser and difficult to locate from the log files.

The hosting company for the server in question eventually resolved the issue when they removed the automatic blacklisting that had been automatically placed on the proxy server’s public IP address after too many people failed to authenticate properly (because the website in question had a password on it).

Here’s the test when it worked properly returning a 401 error requiring the user to authenticate;

SG800#test http get http://someurlsomewhere.com

Type escape sequence to abort.
Executing HTTP get test

* HTTP request header sent:
GET http://someurlsomewhere.com HTTP/1.0
Host: someurlsomewhere.com
User-Agent: HTTP_TEST_CLIENT

* HTTP response header recv'd:
HTTP/1.1 401 Authorization Required
Date: Wed, 11 Aug 2010 15:40:31 GMT
Server: Apache/2.2.3 (CentOS)
WWW-Authenticate: Basic realm="For Acme Health only..."
Content-Type: text/html; charset=iso-8859-1
Cache-Control: proxy-revalidate
Content-Length: 479
Connection: close
Proxy-support: Session-based-authentication

Measured throughput rate is 4.50 Kbytes/sec
HTTP get test passed

Cheers!

Internet Utilization at 99.9% Arrgghhh!

Michael McNamara — Thu, 25 Jun 2009 00:00:31 +0000

I thought I would just share this short story with you all… it’s a classic case of what can happen even with the best of plans and intentions. We recently deployed Adobe Acrobat Reader 9.1.2 via Microsoft Active Directory Group Policy.

We rushed the deployment in order to address some of the recent Acrobat vulnerabilities that were being actively exploited in the wild by Nine-Ball and other trojans/malware. We noticed an unusual uptick in Internet utilization almost immediately after the package had been deployed. When we examined our Websense logs we found an extreme number of HTTP requests to swupd.adbobe.com. We determined that these requests were coming from Adobe software products that were attempting to check for an update via Adobe’s auto-update feature. The HTTP requests were being denied by our Blue Coat ProxySG appliances because we require user authentication to access the Internet. While the Adobe auto-update component was able to read the PAC file configured within Internet Explorer it was not able to provide authentication when challenged with a 407 response. We originally thought the sheer number of clients making requests was putting an undo burden on the system so we added some CPL code to our Blue Coat ProxySG appliances to allow non-authenticated access to *.adobe.com. Within minutes of that change the wheels on the bus came flying off literally.

We just happen to have two 50Mbps Ethernet links to the Internet being served from two Blue Coat ProxySG appliances with about 5,500 client PCs. Within minutes both ProxySG appliances went to 96% CPU utilization and both Internet links went to 99.9% utilization. We had literally let the cat out of the bag and it was off and running… the number of client PCs trying to download updates from Adobe surged and they literally started to choke our two Internet connections.

Thankfully the Blue Coat ProxySG appliances support bandwidth classes. We created a 1Mbps class and added some CPL code to bandwidth restrict access to *.adobe.com. While that proved to be the quick fix we’re also deploying an update via Group Policy to disable the auto-update feature per Adobe’s knowledgebase article.

Cheers!

BlueCoat ProxySG – Flush DNS and Cache

Michael McNamara — Thu, 02 Oct 2008 22:00:42 +0000

There can be a few occasions where you may need to manually purge the local DNS cache and/or the actual web cache of a Blue Coat ProxySG appliance. While both the DNS cache and web cache will eventually age out it can be helpful to sometimes speed up the process by flushing/purging the DNS and web cache.

While this can all be done from the web interface I generally prefer the CLI (if available). The Blue Coat ProxySG appliances that I managed are setup for SSH access you may need to confirm that SSH is enabled (telnet might be enabled).

Let’s start by connecting to the BlueCoat ProxySG appliance (proxysg.acme.org);

[root@linuxhost etc]# ssh -l admin proxysg.acme.org
admin@proxysg.acme.org's password:

proxysg.acme.org - Blue Coat SG510 Series>

Once we’re connected we need to go into privledged mode to issue the commands;

proxysg.acme.org - Blue Coat SG510 Series>enable
Enable Password:

Now that we’re in privledged mode we can clear the web content cache with the following command;

proxysg.acme.org - Blue Coat SG510 Series#clear-cache
ok

And to clear the DNS cache we can use the following command;

proxysg.acme.org - Blue Coat SG510 Series#purge-dns-cache
ok

And don’t forget to logout when you’re all done.

proxysg.acme.org - Blue Coat SG510 Series#exit
Connection to proxysg.acme.org closed.

Cheers!

Blue Coat ProxySG Appliances Load Balancing & High Availability

Michael McNamara — Sun, 10 Aug 2008 19:00:14 +0000

We just recently deployed four Blue Coat ProxySG 510 appliances along with two Blue Coat ProxyAV 810 appliances in our enterprise. The Blue Coat ProxySG appliance is essentially a proxy server that integrates with NTLM/AD authentication and content filtering. The Blue Coat ProxyAV appliance is essentially an Anti-Virus solution, you choose which AV engine to license from Kaspersky, Sophos, McAfee or Panda, for all Internet content.

We have two Internet links located at different geographically locations. We placed two Blue Coat ProxySG 510 appliances at one location along with a single Blue Coat ProxyAV 810 appliance. We setup each ProxySG with it’s own VIP and made each box a backup to the other. We duplicated that same configuration at the second geographic location with two Blue Coat ProxySG 510s and a single Blue Coat ProxyAV 810. The folks that are familiar with the Blue Coat product line might think that the 510 model is probably somewhat undersized for the size of our environment but our design was to use all four 510s in an active/active configuration as opposed to an active/standby configuration. In the event of a appliance failure we may have some high utilization but we won’t suffer an outage.

How did we achieve the load balancing and high availability?

It’s not really that difficult, although let’s not tell the folks writing the checks that. You’ll need to use a PAC file with some special JavaScript code to help provide the poor mans load balancing. The VIPs will provide the redundancy within each geographic site and we’ll trust the browser to provide the redundancy between the two geographic sites.

//
// Blue Coat Proxy Auto-Configuration File (PAC)
//
// This PAC file will provide a poor mans load balancing solution by
// hasing the IP address of the client to help spread the load of the
// client across two different ProxySG appliances. The variables at
// the top of the PAC file need to be adjusted to the specific VIP
// addresses of each ProxySG appliance.
//
function FindProxyForURL(url, host)
{

    // Declare Variables
    var OLDPROXY = "PROXY 10.1.20.100";

    var HECTOR = "PROXY 10.1.127.60";
    var FERGAL = "PROXY 10.1.127.65";
    var MEG = "PROXY 10.2.127.60";
    var SEAN = "PROXY 10.2.127.65";

    var SITE1VIP1 = "PROXY 10.1.127.62";
    var SITE1VIP2 = "PROXY 10.1.127.63";
    var SITE2VIP1 = "PROXY 10.2.127.62";
    var SITE2VIP2 = "PROXY 10.2.127.63";

    var ipSubs = myIpAddress().split(".");
    var localIP = myIpAddress();

    ret = URLhash(url);

    // Use DIRECT if the destination is just a plain WINS name
    if (isPlainHostName (host) ||
       // Use DIRECT if the host is within the michaelfmcnamara.com domain space
       dnsDomainIs(host, ".michaelfmcnamara.com") ||
       // Use DIRECT if the host is within the RFC1918 ten-dot-zero address space
       isInNet(host, "10.0.0.0", "255.255.0.0") ||
    return "DIRECT"; 

    // Use DIRECT if the URL starts with 10.*
    if (shExpMatch(url, "http://10.*") ||
       shExpMatch(url, "https://10.*"))
    return "DIRECT";

   // Use the Blue Coat Proxy for an remaining URL matching "http:"
   if( url.substring(0, 5) == "http:" ) {
      if (isInNet(myIpAddress(), "10.1.0.0", "255.255.0.0") {
         if ( (ipSubs[3] % 2) == 0 ) {
            return SITE1VIP1 + ":8080";
         } else {
            return SITE1VIP2 + ":8080";
         }
      } else {
         if ( (ipSubs[3] % 2) == 0 ) {
            return SITE2VIP1 + ":8080";
         } else {
            return SITE2VIP2 + ":8080";
         }
      }
   } else if( url.substring(0, 6) == "https:" )
      if (isInNet(myIpAddress(), "10.1.0.0", "255.255.0.0") {
         if ( (ipSubs[3] % 2) == 0 ) {
            return SITE1VIP1 + ":8080";
         } else {
            return SITE1VIP2 + ":8080";
         }
      } else {
         if ( (ipSubs[3] % 2) == 0 ) {
            return SITE2VIP1 + ":8080";
         } else {
            return SITE2VIP2 + ":8080";
         }
      }
   } else if( url.substring(0, 5) == "ftp:" ) {
      if (isInNet(myIpAddress(), "10.1.0.0", "255.255.0.0") {
         if ( (ipSubs[3] % 2) == 0 ) {
            return SITE1VIP1 + ":8080";
         } else {
            return SITE1VIP2 + ":8080";
         }
      } else {
         if ( (ipSubs[3] % 2) == 0 ) {
            return SITE2VIP1 + ":8080";
         } else {
            return SITE2VIP2 + ":8080";
         }
      }
   } else if( url.substring(0, 5) == "mms:" ) {
      if (isInNet(myIpAddress(), "10.1.0.0", "255.255.0.0") {
         if ( (ipSubs[3] % 2) == 0 ) {
            return SITE1VIP1 + ":1775";
         } else {
            return SITE1VIP2 + ":1775";
         }
      } else {
         if ( (ipSubs[3] % 2) == 0 ) {
            return SITE2VIP1 + ":1775";
         } else {
            return SITE2VIP2 + ":1775";
         }
      }
   } else {
      return "DIRECT";
   }
}

// This line is just for testing; you can ignore it.  But, if you are having
// problems where you think this PAC file isn't being loaded, then change this
// to read "if (1)" and the alert box should appear when the browser loads this
// file.
//
// This works for IE4, IE5, IE5.5, IE6 and Netscape 2.x, 3.x, and 4.x.
// This does not work for Mozilla (and probably not for Netscape 6.x).
// (For IE6, tested on Win2K)
//
if (0) {
    alert("DEBUG: PAC file LOADED");
}

function URLhash(name)
{
var  cnt=0;
    var str=name.toLowerCase(name);
    if ( str.length ==0) {
         return cnt;
    }
    for(var i=0;i < str.length ; i++) {
       var ch= atoi(str.substring(i,i + 1));
        cnt = cnt + ch;
    }

    return cnt ;
}

/* 
   URLhash2( ) for directory name hash computing version.
   written by SHARP Corp in Feb 1997
   
   Objects in a same directory will be accessed via the same proxy.
   Use URLhash2( ) instead of URLhash( ) if you prefer to use persistent
   connection in HTTP 1.1

   http://www.sharp.co.jp/sample/test/img/mebius.png
   http://www.sharp.co.jp/sample/test/img/zaurus.png
   http://www.sharp.co.jp/sample/test/img/wiz.png
   <------------------------------------->
         directory name hashing here
   
*/

function URLhash2(name)
{
var  cnt=0;
var  dirptr=0;

    var str=name.toLowerCase(name);
    if ( str.length ==0) {
         return cnt;
    }

/* skip filename in directory */
    for(var i=str.length - 1;i >=0 ; i--) {
        if ( str.substring(i,i +1) == '/' ) {
            dirptr = i+1 ;
            break;
        }
    }

    for(var i=0;i < dirptr; i++) {
       var ch= atoi(str.substring(i,i + 1));
        cnt = cnt + ch;
    }

    return cnt ;
}

function atoi(charstring)
{

    if ( charstring == "a" ) return 0x61; if ( charstring == "b" ) return 0x62;
    if ( charstring == "c" ) return 0x63; if ( charstring == "d" ) return 0x64;
    if ( charstring == "e" ) return 0x65; if ( charstring == "f" ) return 0x66;
        if ( charstring == "g" ) return 0x67; if ( charstring == "h" ) return 0x68;
    if ( charstring == "i" ) return 0x69; if ( charstring == "j" ) return 0x6a;
    if ( charstring == "k" ) return 0x6b; if ( charstring == "l" ) return 0x6c;
    if ( charstring == "m" ) return 0x6d; if ( charstring == "n" ) return 0x6e;
    if ( charstring == "o" ) return 0x6f; if ( charstring == "p" ) return 0x70;
    if ( charstring == "q" ) return 0x71; if ( charstring == "r" ) return 0x72;
    if ( charstring == "s" ) return 0x73; if ( charstring == "t" ) return 0x74;
    if ( charstring == "u" ) return 0x75; if ( charstring == "v" ) return 0x76;
    if ( charstring == "w" ) return 0x77; if ( charstring == "x" ) return 0x78;
    if ( charstring == "y" ) return 0x79; if ( charstring == "z" ) return 0x7a;
    if ( charstring == "0" ) return 0x30; if ( charstring == "1" ) return 0x31;
    if ( charstring == "2" ) return 0x32; if ( charstring == "3" ) return 0x33;
    if ( charstring == "4" ) return 0x34; if ( charstring == "5" ) return 0x35;
    if ( charstring == "6" ) return 0x36; if ( charstring == "7" ) return 0x37;
    if ( charstring == "8" ) return 0x38; if ( charstring == "9" ) return 0x39;
    if ( charstring == "." ) return 0x2e;
    return 0x20;
}

Let me provide some explanation of the PAC file attached above. First you may notice that I have the function URLhash and URLhash2 included in the PAC file. I don’t actually use those functions although I may in the future.

There are two geographic sites (10.1.0.0/16 and 10.2.0.0/16) each with two Blue Coat ProxySG appliances and a single ProxyAV appliance. Overall there are four ProxySG appliances and two ProxyAV appliances. The PAC file will essentially route all requests “statically” to the local ProxySG appliances with the script load balancing the client requests, depending on the client’s IP address, to one of the two local ProxySG appliances. If you wanted geographic redundancy you could alter the “return” statement like so;

"PROXY 10.1.127.62:8080; 10.2.127.62:8080"

In this case the browser will attempt to contact the proxy server at 10.1.127.62:8080 and if the browser is unable to contact that proxy it will then try the proxy server at 10.2.127.62:8080. You’ll need to make sure that you include the (semicolon) “;” between each proxy server.

This solution has been working really well. The Blue Coat ProxySG appliance has a really impressive policy and rule base. You can do almost anything you want including coaching pages and warning banners. We current have our Blue Coat ProxySG appliances running Websense (onbox) for content filtering.

Cheers!