My home network lab has two IdM servers that run as replicas. In a perfect world, I’d have at least three so they could vote to overrule an errant replica, but this isn’t practical for me at this time. One of these IdM servers is physical, and the other virtual, so, having upgraded the virtual host to CentOS 8 and experiencing some minor issues, the virtual IdM server ended up offline quite a bit. Enough, it seems, for their clocks to drift apart.
As these are the only two chrony servers configured, as they drifted apart, they began to distrust the others. And, having no other node to back up one’s claim of what time it was, they began to distrust themselves. It’s not obvious this is what’s happening from the behavior I experienced.
I first identified the issue when trying to add a server to the domain. The
ipa-client-install command would contact the IPA servers just fine, but time
synchronization would always time out. Checking chronyc sources on other hosts
would show that the servers were “unreachable” - signified by “^?”. I started
investigating firewalls, taking tcpdumps, and trying to diagnose this as a
network issue.
This was a little bit silly for two reasons: first, time synchronization worked
as of a few days ago when I first upgraded the IdM servers to CentOS 8 and I
hadn’t changed anything relating to /etc/chrony.conf; and secondly, the
servers showed even themselves as unreachable. Though I thought this strange,
I pressed on with my investigation of a possible network issue until exhausting
my will to continue that pursuit. In desparation, I thought “if I can’t get them
to talk to each other, at least maybe I can get them to standardize on a
third-party”.
A touch of server time.nist.gov in /etc/chrony.conf and the servers begin
communcating with NIST’s mirrors. Then themselves. Then each other. Then they
started responding to the rest of the network, bringing all nodes back into sync.
I’ll admit I’ve had trouble understanding fairly basic things about chrony, but
this behavior is just plain cryptic.