Why clocks drift and how to cope
Open in the interactive app: practice, quiz, mock interview →In one minute
- Every machine has its own quartz clock, and quartz drifts, typically 10 to 100 parts per million. Unsynchronised clocks diverge by seconds per day.
- Protocols like NTP and PTP estimate the offset to a reference clock by timing a round trip, and correct the local clock gradually.
- The uncertainty never reaches zero. Expect single-digit milliseconds over a WAN with NTP, tens to hundreds of microseconds in a good datacentre, and sub-microsecond only with hardware timestamping.
- Wall clocks can jump backwards. Use monotonic clocks for durations and never use wall time alone to order events across machines.
How it works
Two clocks in every machine
A time-of-day clock returns the current date and time. NTP may step it, so it can jump forward or backward. A monotonic clock only moves forward and measures elapsed time; its absolute value is meaningless across machines. Timeouts, leases and latency measurements must use the monotonic clock. Timestamps that leave the machine use time-of-day.
Drift and skew
A crystal at 50 ppm drift gains or loses 50 microseconds per second, which is 4.3 seconds per day. Temperature changes drift further. Skew is the difference between two clocks at one instant. Synchronisation keeps skew bounded by measuring and correcting drift continuously.
How NTP estimates offset
The client records t1 and sends a request. The server records receive time t2 and send time t3. The client records arrival t4. Then round-trip delay is (t4 - t1) - (t3 - t2) and estimated offset is ((t2 - t1) + (t3 - t4)) / 2. The formula assumes the outbound and return paths take equal time. If one direction takes 30 ms and the other 10 ms, the estimate is off by 10 ms, and the client cannot detect it. The error is at most half the round-trip delay, so clients sample several servers repeatedly and trust the samples with the lowest delay (RFC 5905).
NTP servers form strata. Stratum 0 is an atomic or GPS reference, stratum 1 servers are attached to it, and each hop adds a stratum and some error. Daemons then slew the clock, running it slightly fast or slow until the offset closes, rather than stepping it. That only holds for small offsets. Classic ntpd slews offsets under 128 ms and steps anything larger, and because the Unix kernel slews at most 0.5 ms per second, each second of error takes about 2,000 s to remove (ntpd docs). chrony does its own slewing at up to 83,333 ppm by default (chrony.conf). So time is monotonic in practice only while offsets stay small; a badly wrong clock at boot, or after a long VM pause, still gets stepped.
Plain NTP packets are unauthenticated, so anyone on the path can shift a client's clock. Network Time Security adds a TLS key exchange and authenticated time packets (RFC 8915, 2020), and chrony enables it per server with the nts option.
Better precision
PTP (IEEE 1588) timestamps packets in the NIC hardware and lets switches correct for their own queueing delay, reaching sub-microsecond skew on a LAN. Meta moved its fleet from NTP to PTP after finding that commit-wait reads ran about 100 times better on PTP-backed clusters; in one 24-hour sample its NTP clocks stayed within about 8 microseconds and PTP within about 100 nanoseconds (Meta). Cloud providers now expose this too. Since November 2023, the Amazon Time Sync Service gives typical error bounds under 100 microseconds over NTP and under 40 microseconds from the PTP hardware clock on supported Nitro instances (AWS).
Leap seconds and smearing
UTC has occasionally inserted a leap second, 23:59:60, most recently at the end of 2016. Software that assumes every minute has 60 seconds has crashed or spun CPU on these days. Large operators instead smear the extra second, but not identically. Google runs a linear smear over 24 hours, noon to noon UTC, making each second about 11.6 ppm long (Google). Meta uses a quadratic smear over 17 hours starting at midnight UTC (Meta). Mixing smeared and non-smeared sources, or two different smears, gives up to about half a second of disagreement.
Leap seconds are now ending. In 2026 the General Conference on Weights and Measures (CGPM) resolved that UTC becomes continuous from 20 May 2027, with no further leap seconds and the UT1-UTC gap allowed to grow to one hour (BIPM, CGPM 2026 Resolution 5). Old timestamps still contain 23:59:60, so parsers must keep handling it, but new systems no longer need a smear plan.
Using clocks safely
Google's Spanner exposes time as an interval [earliest, latest] with a known error bound, typically a few milliseconds (a sawtooth from about 1 to 7 ms in the published design), and waits out that uncertainty before committing so timestamps respect real-time order (Spanner paper). Without such a bound, treat wall-clock timestamps from different machines as approximate.
Down the stack
Layer 1The contract
Linux exposes several clocks through clock_gettime, each with a different promise. CLOCK_REALTIME is UTC-ish wall time and can be stepped in either direction. CLOCK_MONOTONIC never goes backwards but is still slewed by NTP, so its rate tracks true seconds. CLOCK_MONOTONIC_RAW is the uncorrected hardware rate. CLOCK_BOOTTIME also counts time spent suspended, and CLOCK_TAI counts leap seconds instead of repeating one (clock_gettime(2)). Each call fills a 16-byte struct timespec of seconds and nanoseconds.
A sync daemon promises a bound, not an exact time. chronyc tracking reports the current offset, root delay and root dispersion, and chrony documents the absolute bound as the offset plus root dispersion plus half the root delay (chronyc docs). TrueTime makes the bound the API: TT.now() returns an interval [earliest, latest] guaranteed to contain true time, plus TT.after(t) and TT.before(t). AWS publishes a similar daemon, ClockBound, that hands applications the current error bound.
Layer 2The mechanism
An NTP packet is a 48-byte UDP datagram on port 123. It carries a 2-bit leap indicator, version, mode, stratum, poll exponent, precision, root delay, root dispersion, a reference ID, and four 64-bit timestamps: 32 bits of seconds since 1900 and 32 bits of fraction, a resolution of about 233 picoseconds. The 32-bit seconds field wraps in 2036, which the protocol handles with era numbers.
Clients poll every 2^poll seconds, by default somewhere between 64 s and 1024 s. Each source's recent samples (eight in classic ntpd) go through a clock filter that prefers the minimum-delay sample, because low delay means little room for asymmetric queueing. An intersection algorithm in the spirit of Marzullo's then compares the sources' confidence intervals and discards falsetickers whose intervals do not overlap the majority, which is why you configure at least four servers. The survivors feed a control loop that adjusts the local frequency and phase.
The daemon applies corrections through adjtimex(2): ADJ_FREQUENCY changes the rate in scaled parts per million and ADJ_OFFSET asks the kernel to slew away an offset. The kernel's own discipline caps slewing at 500 ppm, so a 1 s error takes over half an hour to remove. chrony bypasses that loop and can slew much faster and uses makestep (for example makestep 1.0 3) to permit a step only during the first few updates after boot.
PTP replaces software guesses with hardware: Sync and Follow_Up carry the master's send time, Delay_Req and Delay_Resp measure the return path, and transparent-clock switches add their residence time into a correctionField.
Layer 3The machine
On x86 the kernel usually reads time from the invariant TSC, a per-core counter running at a constant rate. Timekeeping converts cycles to nanoseconds as (cycles * mult) >> shift, and NTP corrections work by nudging mult. Through the vDSO, clock_gettime reads that counter without entering the kernel and costs on the order of 20 ns. If the clocksource falls back to HPET or ACPI PM, common in some VMs and visible in /sys/devices/system/clocksource/clocksource0/current_clocksource, every call becomes a real syscall plus slow I/O and can cost a microsecond or more.
Software timestamps are taken after the NIC raises an interrupt, and interrupt coalescing alone can add tens of microseconds of jitter. That is the floor for software NTP inside a datacentre. Hardware timestamping (SO_TIMESTAMPING) stamps packets in the NIC's PTP hardware clock, exposed as /dev/ptp0, and phc2sys then disciplines the system clock from it, which is how sub-microsecond skew is reached.
Spanner's published design polls GPS and atomic time masters every 30 s and assumes a worst-case local drift of 200 us per second between polls, so uncertainty grows in a sawtooth from about 1 ms to about 7 ms and resets after each poll. The two reference types are deliberate: GPS fails through antennas, interference and spoofing, atomic clocks through slow drift, and the failures rarely coincide.
Why it's built this way
- Quartz frequency depends on cut and temperature, so every clock drifts and must be measured against a reference continuously.
- Measurement crosses a network whose two directions are not guaranteed symmetric, so offset can only be estimated within half the round trip, and low-delay samples are worth the most.
- Time going backwards breaks software, so corrections are slewed rather than stepped, which makes convergence slow.
- Since error never reaches zero, the honest interface is an interval, and systems that need ordering either wait out the interval or stop depending on physical time.
Diagram
sequenceDiagram
participant C as Client
participant S as "NTP server"
Note over C: t1 = send time (client clock)
C->>S: request
Note over S: t2 = receive time (server clock)
Note over S: t3 = reply time (server clock)
S-->>C: reply with t2, t3
Note over C: t4 = arrival (client clock)
Note over C: delay = (t4-t1) - (t3-t2)
Note over C: offset = ((t2-t1)+(t3-t4))/2Worked example
Client sends at t1 = 100.000. Server receives at t2 = 100.120 and replies at t3 = 100.121. Client receives at t4 = 100.041, all in seconds.
Delay is (100.041 - 100.000) - (100.121 - 100.120) = 0.041 - 0.001 = 0.040, so 40 ms on the wire. Offset is ((0.120) + (0.080)) / 2 = 0.100. The client is 100 ms behind the server, with uncertainty up to plus or minus 20 ms if the paths were asymmetric.
Now the consequence. A system uses last-write-wins with wall-clock timestamps. Node A is 100 ms behind. A user writes status=away on node B at true time 12:00:00.050 (stamped .050), then status=online on node A at true time 12:00:00.120 (stamped .020). LWW keeps away, the older write, and silently drops the newer one. With 3 replicas and 2,000 writes per second to hot keys, any skew larger than the gap between conflicting writes loses data.
In the wild
- Linux fleets, 2012: the leap second livelocks futex users. After the 30 June 2012 leap second, many Linux machines hit load spikes in MySQL, Java, Firefox and Thunderbird. The kernel applied the extra second without telling the high-resolution timer subsystem, so
CLOCK_REALTIMEtimers fired a second early and sub-second timeouts fired immediately, in a tight loop. The workaround was to set the clock to itself (date -s "$(date)"), and John Stultz's patches made leap-second handling callclock_was_set()(LKML). - Cloudflare, 2017: time goes backwards in RRDNS. At midnight UTC on 1 January the leap second stepped the wall clock back, Go's then wall-clock
time.Now()produced negative round-trip times, and a negative value reachingrand.Int63n()panicked the DNS proxy. At peak about 0.2% of DNS queries failed. Cloudflare discarded negative intervals and audited other duration code; Go 1.9 later added monotonic readings totime.Time(Cloudflare, Go 1.9 notes).
Trade-offs
| Approach | Strengths | Weaknesses | Choose when |
|---|---|---|---|
| NTP over internet | Free, universal | Milliseconds of error; asymmetric paths | General servers, logging |
| NTP or chrony to local stratum 1 | Sub-millisecond in datacentre | Needs internal time infrastructure | Most production fleets |
| PTP with hardware timestamps | Sub-microsecond | Special NICs and switches | Trading, telecoms, tight ordering |
| Bounded-uncertainty time (TrueTime-style) | Safe real-time ordering | Commit wait; custom infrastructure | Externally consistent databases |
| Logical clocks | No physical sync needed | No relation to real time | Causal ordering of events |
Pitfalls in production
- Measuring durations with the wall clock, which goes negative when NTP steps backwards. Fix: use monotonic clock APIs for every timeout and latency measurement.
- Last-write-wins on wall time across nodes. Fix: use hybrid logical clocks or version vectors, or accept and document the loss window.
- A VM pause or live migration causing a jump of several seconds. Fix: alert on clock offset, and design leases with margin for pauses.
- No monitoring of offset. Fix: export chrony or NTP offset as a metric and alert above a threshold such as 50 ms, and fence nodes far out of sync.
Leadership lens
- Ask which correctness properties depend on clocks, and what skew bound they assume. If nobody knows the bound, the design is not safe.
- Treat time infrastructure as a production dependency with owners, monitoring and runbooks.
- Prefer designs that need only monotonic local time plus logical ordering. Use synchronised physical time for humans and for bounded-error tricks you can prove.
Check yourself
Why can NTP not measure a one-way network delay?
Show answer
It only sees four timestamps from two different clocks. Offset and path asymmetry are indistinguishable, so it assumes symmetric paths and bounds the error by half the round trip.
Why should a lease timer use the monotonic clock?
Show answer
The wall clock can be stepped forward or backward by synchronisation, which would make a lease appear to expire early or last too long.
What does slewing do that stepping does not?
Show answer
It corrects the offset by running the clock slightly fast or slow, so time never jumps and never goes backwards.
Sources and further study
Primary
- RFC 5905: Network Time Protocol Version 4: the packet format, clock filter, selection and discipline algorithms.
- RFC 8915: Network Time Security for NTP: authenticated NTP with TLS key establishment.
- Corbett et al., Spanner (OSDI 2012): TrueTime, its error bound and commit wait.
- BIPM, CGPM 2026 Resolution 5: the decision to stop leap seconds from 20 May 2027.
Systems docs
- chrony.conf reference:
makestep,maxslewrate, poll intervals and NTS options. - Linux clock_gettime(2): what each kernel clock promises.
- Amazon Time Sync Service and the PTP hardware clock (AWS): microsecond bounds on EC2 and ClockBound.
- Precision Time Protocol at Meta: why a large fleet moved from NTP to PTP.
Books
- Designing Data-Intensive Applications, Martin Kleppmann, chapter 8 "The Trouble with Distributed Systems" (1st edition), section "Unreliable Clocks": monotonic versus time-of-day clocks, LWW data loss and confidence intervals.
- Understanding Distributed Systems, Roberto Vitillo, chapter "Time": physical clocks, NTP and why they cannot order events.
Other curricula
- Kleppmann, Distributed Systems lecture notes (Cambridge): sections 3.1 and 3.2 cover quartz drift, leap seconds and NTP's slew, step and panic thresholds.
- donnemartin/system-design-primer: does not cover clocks or time synchronisation at all, which is why it pairs poorly with this topic.
Watch
- Distributed Systems 3.1: Physical time (Martin Kleppmann): quartz drift, UTC, leap seconds and clock types.
- Distributed Systems 3.2: Clock synchronisation (Martin Kleppmann): how NTP estimates offset and corrects clocks.
Free resources
- Network Time Protocol (Wikipedia): strata, the offset formula and protocol history.
- Building a more accurate time service at Facebook scale (Meta Engineering): moving a large fleet to chrony and measuring accuracy.
- Leap smear (Google Public NTP): how and why Google smears leap seconds.
- Spanner: TrueTime and external consistency (Google Cloud): using bounded clock uncertainty for ordering.
- How and why the leap second affected Cloudflare DNS (Cloudflare): a real outage caused by time going backwards.