I’m going to make a series of posts around the A10 load balancers over the next few weeks to try and highlight some configuration options as I start working on some of my larger migrations and configurations.

In this post I’ll highlight just simple load-balancing. We won’t worry about URL paths (Layer 7), persistence, stickiness, cookies, SSL termination or any of those exciting topics today. I’ll be working in a partition called “PART-TEST” although you can setup whatever partition you’d like on either your physical or virtual ADCs.

active-partition PART-TEST
vlan 10
 untagged ethernet 2
 router-interface ve 10

interface ve 10
 ip address 10.1.1.250 255.255.255.0
!
ip route 0.0.0.0 /0 10.1.1.1

In the example above we setup VLAN 10 and bound it to interface Ethernet 2 and bond that to the IP address of 10.1.1.250 adding a static default route to 10.1.1.1.

ip nat pool PART-TEST_SRCNAT 10.11.1.240 10.1.1.249 netmask /24

We’ll using source NAT so we can return the traffic to the loader. The downside of this is that we loose the real source IP address of the client, the upside is that we gain a lot of flexibility in that we don’t need traffic to “route” through the load balancer and we don’t need Layer 2 adjacency between the load balancer and the real servers.

health monitor SIMPLE_HTTP_PROBE
 method http url GET /health expect response-code 200

We’ll setup a simple HTTP probe that will hit the /health path and look for a HTTP/200 code.

slb server RS_SERVER1 10.1.1.51
no health-check
port 80 tcp

slb server RS_SERVER2 10.1.1.52
no health-check
port 80 tcp

We’ll setup two real servers RS_SERVER1 (10.1.1.51) and RS_SERVER2 (10.1.1.52) and enable TCP port 80 on each of them.

slb service-group SF_GROUP1 tcp
    health-check SIMPLE_HTTP_PROBE
    member RS_SERVER1:80
    member RS_SERVER2:80

We’ll add a service group which will allow us to bind multiple real servers into a group. We’ll add our health check at this level. We don’t want or need health checks at too many points, we’re just generating a lot of noise on the wire and burning CPU cycles on the ADC.

slb virtual-server VIP_GROUP1 10.1.1.150
   port 80  tcp
      name SLB_GROUP1-HTTP
      source-nat pool PART-TEST_SRCNAT
      service-group SF_SGROUP1

And here’s where we tie everything together. We create a VIP (virtual server), assign an IP address, add a port and then bind the service group we previously created along with the source NAT pool.

If you’re curious and want to play around you can quickly setup your own ADC, utilizing a 30 day trial version vThunder. The vThunder is supported on VMware ESXi, Microsoft Hyper-V, KVM and XenServer. If your an Amazon Web Services (AWS) customer you can also leverage the vThunder in that environment.

Cheers!

Credits: Image by Mario

slb service-group SG_STORE-HTTP tcp
 member RS_STORE-WEB1:8080
 member RS_STORE-WEB2:8080
 member RS_STORE-WEB3:8080

While there is no “inservice standby” command you can change the priority of the member which will provide the equivalent result on the real server in the service-group. The higher the value the more preferred a real server.

slb service-group SG_STORE-HTTP tcp
 member RS_STORE-WEB1:8080
 member RS_STORE-WEB2:8080 priority 16
 member RS_STORE-WEB3:8080 priority 16

Here’s what the CLI reference guide says about he member priority command;

Primary and backup servers are designated based on member priority (set with the member command). For example, if a service group contains real servers with the following priority settings, real servers s1, s2, and s3 are the primary servers. Real servers s4 and s5 are backup servers.
• s1 – priority 16
• s2 – priority 16
• s3 – priority 16
• s4 – priority 8
• s5 – priority 8
When the minimum number of active members (primary servers) comes back up, the AX device immediately returns to using only the primary servers.

Cheers!
Note: This is a series of posts made under the Network Engineer in Retail 30 Days of Peak, this is post number 9 of 30. All the posts can be viewed from the 30in30 tag.

When a web browser opens a connection to a web server it will generally send multiple requests across a single TCP connection similar to the figure below. Occasionally some browsers will even utilize HTTP pipelining if both the server and browser support that feature, sending multiple requests without waiting for the corresponding TCP acknowledgement.

The majority of load balancers, including the Cisco ACE 4710 and the A10 AX ADC/Thunder, will look at the first request in the TCP connection and apply the load-balancing metric and forward the traffic to a specific real server in the VIP. In order to speed the processing of future requests the load balancer will forward all traffic in that connection to the same real server in the VIP. This generally isn’t a problem if there’s only a single user associated with a TCP connection.

Akamai will attempt to optimize the number of TCP connections from their edge servers to your origin web servers by sending multiple requests from different users all over the same TCP connection. In the example below there are requests from three different users but it’s been my experience that you could see requests for dozens or even hundreds of users across the same TCP connection.

And here lies the problem, the load balancer will only evaluate the first request in the TCP connection, all subsequent requests will be sent to the same real server leaving some servers over utilized and others under utilized.

Thankfully there are configuration options in the majority of load balancers to work around this problem and instruct the load balancer to evaluate all requests in the TCP connection independently.

A10 AX ADC/Thunder

strict-transaction-switch

Cisco ACE 4710

parameter-map type http HTTP_PARAMETER_MAP
  persistence-rebalance strict

With the configuration change made now every request in the TCP connection is evaluated and load-balanced independently resulting in a more even distribution across the real servers in the farm.

In this scenario I’m using HTTP cookies to provide session persistence and ‘stickiness’ for the user sessions. If your application is stateless then you don’t really need to worry that a user lands on the same real server for each and every request.

Cheers!

Image Credit: topfer

We have two Internet links located at different geographically locations. We placed two Blue Coat ProxySG 510 appliances at one location along with a single Blue Coat ProxyAV 810 appliance. We setup each ProxySG with it’s own VIP and made each box a backup to the other. We duplicated that same configuration at the second geographic location with two Blue Coat ProxySG 510s and a single Blue Coat ProxyAV 810. The folks that are familiar with the Blue Coat product line might think that the 510 model is probably somewhat undersized for the size of our environment but our design was to use all four 510s in an active/active configuration as opposed to an active/standby configuration. In the event of a appliance failure we may have some high utilization but we won’t suffer an outage.

How did we achieve the load balancing and high availability?

It’s not really that difficult, although let’s not tell the folks writing the checks that. You’ll need to use a PAC file with some special JavaScript code to help provide the poor mans load balancing. The VIPs will provide the redundancy within each geographic site and we’ll trust the browser to provide the redundancy between the two geographic sites.

//
// Blue Coat Proxy Auto-Configuration File (PAC)
//
// This PAC file will provide a poor mans load balancing solution by
// hasing the IP address of the client to help spread the load of the
// client across two different ProxySG appliances. The variables at
// the top of the PAC file need to be adjusted to the specific VIP
// addresses of each ProxySG appliance.
//
function FindProxyForURL(url, host)
{

    // Declare Variables
    var OLDPROXY = "PROXY 10.1.20.100";

    var HECTOR = "PROXY 10.1.127.60";
    var FERGAL = "PROXY 10.1.127.65";
    var MEG = "PROXY 10.2.127.60";
    var SEAN = "PROXY 10.2.127.65";

    var SITE1VIP1 = "PROXY 10.1.127.62";
    var SITE1VIP2 = "PROXY 10.1.127.63";
    var SITE2VIP1 = "PROXY 10.2.127.62";
    var SITE2VIP2 = "PROXY 10.2.127.63";

    var ipSubs = myIpAddress().split(".");
    var localIP = myIpAddress();

    ret = URLhash(url);

    // Use DIRECT if the destination is just a plain WINS name
    if (isPlainHostName (host) ||
       // Use DIRECT if the host is within the michaelfmcnamara.com domain space
       dnsDomainIs(host, ".michaelfmcnamara.com") ||
       // Use DIRECT if the host is within the RFC1918 ten-dot-zero address space
       isInNet(host, "10.0.0.0", "255.255.0.0") ||
    return "DIRECT"; 

    // Use DIRECT if the URL starts with 10.*
    if (shExpMatch(url, "http://10.*") ||
       shExpMatch(url, "https://10.*"))
    return "DIRECT";

   // Use the Blue Coat Proxy for an remaining URL matching "http:"
   if( url.substring(0, 5) == "http:" ) {
      if (isInNet(myIpAddress(), "10.1.0.0", "255.255.0.0") {
         if ( (ipSubs[3] % 2) == 0 ) {
            return SITE1VIP1 + ":8080";
         } else {
            return SITE1VIP2 + ":8080";
         }
      } else {
         if ( (ipSubs[3] % 2) == 0 ) {
            return SITE2VIP1 + ":8080";
         } else {
            return SITE2VIP2 + ":8080";
         }
      }
   } else if( url.substring(0, 6) == "https:" )
      if (isInNet(myIpAddress(), "10.1.0.0", "255.255.0.0") {
         if ( (ipSubs[3] % 2) == 0 ) {
            return SITE1VIP1 + ":8080";
         } else {
            return SITE1VIP2 + ":8080";
         }
      } else {
         if ( (ipSubs[3] % 2) == 0 ) {
            return SITE2VIP1 + ":8080";
         } else {
            return SITE2VIP2 + ":8080";
         }
      }
   } else if( url.substring(0, 5) == "ftp:" ) {
      if (isInNet(myIpAddress(), "10.1.0.0", "255.255.0.0") {
         if ( (ipSubs[3] % 2) == 0 ) {
            return SITE1VIP1 + ":8080";
         } else {
            return SITE1VIP2 + ":8080";
         }
      } else {
         if ( (ipSubs[3] % 2) == 0 ) {
            return SITE2VIP1 + ":8080";
         } else {
            return SITE2VIP2 + ":8080";
         }
      }
   } else if( url.substring(0, 5) == "mms:" ) {
      if (isInNet(myIpAddress(), "10.1.0.0", "255.255.0.0") {
         if ( (ipSubs[3] % 2) == 0 ) {
            return SITE1VIP1 + ":1775";
         } else {
            return SITE1VIP2 + ":1775";
         }
      } else {
         if ( (ipSubs[3] % 2) == 0 ) {
            return SITE2VIP1 + ":1775";
         } else {
            return SITE2VIP2 + ":1775";
         }
      }
   } else {
      return "DIRECT";
   }
}

// This line is just for testing; you can ignore it.  But, if you are having
// problems where you think this PAC file isn't being loaded, then change this
// to read "if (1)" and the alert box should appear when the browser loads this
// file.
//
// This works for IE4, IE5, IE5.5, IE6 and Netscape 2.x, 3.x, and 4.x.
// This does not work for Mozilla (and probably not for Netscape 6.x).
// (For IE6, tested on Win2K)
//
if (0) {
    alert("DEBUG: PAC file LOADED");
}

function URLhash(name)
{
var  cnt=0;
    var str=name.toLowerCase(name);
    if ( str.length ==0) {
         return cnt;
    }
    for(var i=0;i < str.length ; i++) {
       var ch= atoi(str.substring(i,i + 1));
        cnt = cnt + ch;
    }

    return cnt ;
}

/* 
   URLhash2( ) for directory name hash computing version.
   written by SHARP Corp in Feb 1997
   
   Objects in a same directory will be accessed via the same proxy.
   Use URLhash2( ) instead of URLhash( ) if you prefer to use persistent
   connection in HTTP 1.1

   http://www.sharp.co.jp/sample/test/img/mebius.png
   http://www.sharp.co.jp/sample/test/img/zaurus.png
   http://www.sharp.co.jp/sample/test/img/wiz.png
   <------------------------------------->
         directory name hashing here
   
*/

function URLhash2(name)
{
var  cnt=0;
var  dirptr=0;

    var str=name.toLowerCase(name);
    if ( str.length ==0) {
         return cnt;
    }

/* skip filename in directory */
    for(var i=str.length - 1;i >=0 ; i--) {
        if ( str.substring(i,i +1) == '/' ) {
            dirptr = i+1 ;
            break;
        }
    }

    for(var i=0;i < dirptr; i++) {
       var ch= atoi(str.substring(i,i + 1));
        cnt = cnt + ch;
    }

    return cnt ;
}

function atoi(charstring)
{

    if ( charstring == "a" ) return 0x61; if ( charstring == "b" ) return 0x62;
    if ( charstring == "c" ) return 0x63; if ( charstring == "d" ) return 0x64;
    if ( charstring == "e" ) return 0x65; if ( charstring == "f" ) return 0x66;
        if ( charstring == "g" ) return 0x67; if ( charstring == "h" ) return 0x68;
    if ( charstring == "i" ) return 0x69; if ( charstring == "j" ) return 0x6a;
    if ( charstring == "k" ) return 0x6b; if ( charstring == "l" ) return 0x6c;
    if ( charstring == "m" ) return 0x6d; if ( charstring == "n" ) return 0x6e;
    if ( charstring == "o" ) return 0x6f; if ( charstring == "p" ) return 0x70;
    if ( charstring == "q" ) return 0x71; if ( charstring == "r" ) return 0x72;
    if ( charstring == "s" ) return 0x73; if ( charstring == "t" ) return 0x74;
    if ( charstring == "u" ) return 0x75; if ( charstring == "v" ) return 0x76;
    if ( charstring == "w" ) return 0x77; if ( charstring == "x" ) return 0x78;
    if ( charstring == "y" ) return 0x79; if ( charstring == "z" ) return 0x7a;
    if ( charstring == "0" ) return 0x30; if ( charstring == "1" ) return 0x31;
    if ( charstring == "2" ) return 0x32; if ( charstring == "3" ) return 0x33;
    if ( charstring == "4" ) return 0x34; if ( charstring == "5" ) return 0x35;
    if ( charstring == "6" ) return 0x36; if ( charstring == "7" ) return 0x37;
    if ( charstring == "8" ) return 0x38; if ( charstring == "9" ) return 0x39;
    if ( charstring == "." ) return 0x2e;
    return 0x20;
}

Let me provide some explanation of the PAC file attached above. First you may notice that I have the function URLhash and URLhash2 included in the PAC file. I don’t actually use those functions although I may in the future.

There are two geographic sites (10.1.0.0/16 and 10.2.0.0/16) each with two Blue Coat ProxySG appliances and a single ProxyAV appliance. Overall there are four ProxySG appliances and two ProxyAV appliances. The PAC file will essentially route all requests “statically” to the local ProxySG appliances  with the script load balancing the client requests, depending on the client’s IP address, to one of the two local ProxySG appliances. If you wanted geographic redundancy you could alter the “return” statement like so;

"PROXY 10.1.127.62:8080; 10.2.127.62:8080"

In this case the browser will attempt to contact the proxy server at 10.1.127.62:8080 and if the browser is unable to contact that proxy it will then try the proxy server at 10.2.127.62:8080. You’ll need to make sure that you include the (semicolon) “;” between each proxy server.

This solution has been working really well. The Blue Coat ProxySG appliance has a really impressive policy and rule base. You can do almost anything you want including coaching pages and warning banners. We current have our Blue Coat ProxySG appliances running Websense (onbox) for content filtering.

Cheers!