On a recent Sunday night, I was home watching The Walking Dead, a colleague called to say that our core switch was down in our colo. As he was on vacation, I raced over there, preformed some troubleshooting, finally rebooting our huge Cisco 4510 to bring everything back online, that is except Exchange.
The Exchange cluster was using iSCSI, thus was offline during the outage and now would not mount the databases; both the System Attendant and Information Store services would not start. After trying a reboot and examining the Failover Clustering, these error messages were found in the Event Log:
System attendant - EventID: 1005 Unexpected error The clocks on the client and server machines are skewed. ID no: 80090324 Microsoft Exchange System Attendant occurred.
MSExchangeIS - EventID: 5003 Unable to initialize the Microsoft Exchange Information Store service because the clocks on the client and on the server machine are skewed. This may be caused by a time change either in the client or the server machine, and may require a reboot of that machine. Other than that, verify that your domain is properly configured and is currently online.
I didn't even consider the fact that the Exchange server's system time didn't match the domain controllers'. Both nodes were a full six minutes behind! Once the clocks were synced with the DCs, both services started, databases mounted and we were back in business.
Due to the fact we're running a hybrid UNIX/Windows DNS network, oddities like this seem to creep up. Once I have this sorted, there will likely be a blog post about that as well.
No comments:
Post a Comment