I stumbled into an interesting issue today that gave me a smile when I determined it was a DNS issue.
I was doing some consulting work around WireGuard for a client, and noticed a number of odd issues and just general wonky behavior with everything being slow. This specific client uses Ubuntu Linux while I’m more of a RedHat/CentOS/Rocky guy so I thought it was an issue with the DNS caching that Ubuntu utilizes in systemd-resolve. A few quick tests using a Windows client proved that the issues weren’t limited to just the Ubuntu server, it was impacting every device. DNS queries were taking between 5 to 6 seconds and some were timing out entirely.
The client had mentioned some oddities and issues and I thought there might be a duplicate IP on the network – pretty standard affair in some networks. This wasn’t a duplicate IP issue so I went straight to the DNS servers themselves – Microsoft Windows Server 2019. I found that the root forwarders for each server were setup to use some very old Verizon DNS servers – and wouldn’t you know that some of them were no longer responding. I removed all the Verizon entries and added the two standard Google DNS servers – 8.8.8.8, 8.8.4.4. After applying that and restarting each DNS server the problem was gone and everything was running smoothly again.
What do you use for your DNS forwarders? Or do you rely on the root hints file maintained by Internic?
Cheers!
We have Cisco Umbrella forwarders in our environment and I hate them. They don’t cache! So when we have thousands of machines keeping tabs on a cloud service, that now equates to thousands of DNS calls leaving our network every minutes rather than just one or two. It’s hammering our firewalls and it’s gross.
I think we all have similar gremlins and/or challenges.. we’ve deployed dnsmasq on our 1000+ Linux VMs to help take the load off our DNS infrastructure and it works pretty well.
Thanks for the comment!
I always (even at home) use at least a Unix DNS cache server (Bind before, Unbound now) with root hints. The windows servers are configured to be only autoritative (recurse in disabled).
Non AD zones are handled by Unix autoritative server. I do not do recursive and autoritative work on the same machine any more to avoid almost all DNS issues and ease debug in case…
That’s pretty straight forward… no Raspberry Pi in there? ;)
Thanks for the comment!