Michael McNamara

technology, networking, virtualization and IP telephony

DNS Loops – how to not configure DNS forwarding

by

I’m continually amazed by how much I don’t know and by all the little issues and problems that I encounter in my day to day tasks.  You never know what’s going to pop-up.

I recently stumbled over a very poorly configured DNS environment and thought I would share how to not configuring DNS forwarding.

There was a standalone Microsoft domain which had four domain controllers which were set to forward their requests to another pair of Microsoft DNS servers which eventually forwarded those requests to a fairly new Infoblox DNS environment. Upon looking at the Infoblox reports I noticed a number of non-resolvable hostnames at the top of the reports, they outpaced the next domains in the list by several million requests. Assuming that there was some mis-configured application server that was continually pounding the DNS environment I decided to hunt through the logs to see if I could identify the original requestor and get them to clean up their act. I enabled query logging on one of the servers and set out to examine who was making the request. Oddly enough I found that the other three DNS servers were making the request. Ok, I went to the next server and repeated the steps finding that the next server showed that again the other three servers were making the request. I repeated the process on the remaining two servers and found that all the requests for this bad hostname that I could capture weren’t coming from a specific client but were instead coming just from the servers themselves. This isn’t completely odd, the servers themselves can be clients at the same time but the volume of requests was huge and there was nothing running on these servers except for being Microsoft Domain Controllers and Domain Name Services for a very small Microsoft environment.

It wasn’t until I opened the Microsoft DNS server configuration that I was able to piece together what was happening.

DNS loop

The servers were configured all as (multiple) Primary Masters for the internal domains but they had all been configured to use each other as forwarders along with OpenDNS. So in short the configuration was causing a DNS loop for any requests that failed to resolve. A query from a client to WEST-02 would be forwarded to all three DNS servers along with OpenDNS. Those three DNS servers would then forward that query again back to each of the other servers over and over and over. There is no TTL on a DNS query with respect to propagation. A DNS query can be propagated through as many servers as needed. There is a TTL value on the actual DNS record but that value is used in determining the caching lifecycle.

As I understand it a single DNS query for a bad DNS name would continually self propagate in this configuration because the servers would continually try to obtain an answer by looking to the forwarders and since the forwarders were configured to forward to each other you’d end up in a loop scenario.

You should never configure a pair of DNS servers to forward to each other.

Cheers!

Image Credit: airfin

ISC BIND 9.10.2-P3 Forwarding Caching Only Nameserver

by

I recently had to migrate a large DNS environment from about 23 Microsoft Domain Controllers to Infoblox DNS. I could have just deleted all the zones and set the forwarding on the Microsoft DNS servers but I wanted to leave the Microsoft DNS configuration and data in place to provide a quick backout option in the unlikely event that it was need (it was needed but the second time around using the named.conf file below was the charm).

PrintI ended up deploying ISC BIND 9.10.2-P3 across a mix of Windows 2003 and Windows 2008 domain controller servers, some 32-bit and some 64-bit.

As I alluded to above I originally had issues running BIND getting error messages such as the following after only a few hours running the service and clients failing to get name resolution.

27-Jul-2015 19:15:04.575 general: error: ..\client.c:2108: unexpected error:
27-Jul-2015 19:15:04.575 general: error: failed to get request's destination: failure
27-Jul-2015 19:15:04.981 general: error: ..\client.c:2108: unexpected error:
27-Jul-2015 19:15:04.981 general: error: failed to get request's destination: failure
27-Jul-2015 19:15:20.971 general: error: ..\client.c:2108: unexpected error:
27-Jul-2015 19:15:20.971 general: error: failed to get request's destination: failure

There were also a few other errors that apeared to be releated to the anti-DDoS mechanisms built into BIND;

27-Jul-2015 19:50:02.369 resolver: notice: clients-per-query increased to 15

So I went back and recrafted the named.conf file and came up with the following which seems to be working well for me now almost 5 days after the Infoblox DNS migration.

You’ll noticed that I commented out the localhost zone and the 127.0.0.1 reverse zone as well. I didn’t think that BIND would run without them but sure enough it does. I also enabled query logging so I could see what type of abuse the DNS servers were getting. I found a couple of servers that were querying more than 40,000 times a minute for a management platform that had been retired almost 5+ years ago.

options {
  directory "c:\program files\isc bind 9\bin";
 
  // here are the servers we'll send all our queries to
  forwarders {10.1.1.1; 10.2.2.2;};
  forward only;

  auth-nxdomain no;

  // need to include allow-query at a minimum
  allow-recursion { "any"; };
  allow-query { "any"; };
  allow-transfer { "none"; };

  // lets leave IPv6 off for now less to worry about
  listen-on-v6 { "none"; };

  // standard stuff
  version none;
  minimal-responses yes;
 
  // cache positive and negative results for only 5 minutes
  max-cache-ttl 300;
  max-ncache-ttl 300;

  // disable DDoS mechanisms in BIND
  clients-per-query 0;
  max-clients-per-query 0;

};

logging{
   channel example_log{
    file "C:\program files\isc bind 9\log\named.log" versions 3 size 250k;
    severity info;
    print-severity yes;
    print-time yes;
    print-category yes;
  };

  channel queries_file {
    file "c:\program files\isc bind 9\log\queries.log" versions 10 size 10m;
    severity dynamic;
    print-time yes;
  };

  category default{ example_log; };
  category queries { queries_file; };

};

//zone "localhost" in{
//  type master;
//  file "pri.localhost";
//  allow-update{none;};
//};

//zone "0.0.127.in-addr.arpa" in{
//  type master;
//  file "localhost.rev";
//  allow-update{none;};
//};

I setup my first nameserver running BIND 4.x back in 1995, more than 20 years ago while working at Manhattan College. While I'm pretty familiar with BIND a lot has changed since then and so I had to-do a fair bit of research to arrive at the configuration above.

Hopefully someone else will find it helpful.

Cheers!

Recent Posts

Categories

SUBSCRIBE BY EMAIL