I recently took on the responsibility of managing and re-building the Windows Server team. Thanks to Microsoft Security Patch KB3002657 last week was a true adventure in both patch management and change control. It was discovered that several Windows Domain Controllers were missing some critical security patches so it was decided, rather haphazardly, to patch those Domain Controllers immediately. You can guess the chaos that shortly ensued afterwards. The catch – the issue wasn’t with the Domain Controllers that needed to be patched but rather with a legacy Windows 2003 Domain Controller (older physical box) that was left to authenticate requests while the other Domain Controllers rebooted. Unknowingly KB3002657 had been applied to this legacy Windows 2003 Domain Controller automatically by WSUS on the prior Thursday but no issues had been detected at that time because the other Domain Controllers in the Site had been authenticating requests for Microsoft Outlook MAC Clients, IIS and SharePoint. We originally suspected the Domain Controllers that had just been patched but quickly ruled them out and were able to determine which Domain Controller was failing to properly authenticate NTLM requests.
It doesn’t help that Windows 2003 Server is End of Support on July 15, 2015.
Cheers!
References;
https://technet.microsoft.com/en-us/library/security/ms15-027.aspx
http://blogs.technet.com/b/rmilne/archive/2015/03/17/update-015_2d00_027-revised-_2800_3002657_2900_.aspx
http://www.infoworld.com/article/2897814/operating-systems/server-2003-admins-beware-microsoft-re-issues-botched-netlogon-patch-kb-3002657.html