Best practices for selecting an external NTP time source
Applies to
- StorageGRID 7.5
- StorageGRID 8.1
- StorageGRID 8.5
- StorageGRID 9.0
Answer
Background:
StorageGRID comprises of a distributed set of services, mostly running on separate hardware. This distributed nature requires a precise and tight timing of the underlying hardware; if the clocks on the servers deviate too much, StorageGRID will sever communication to the outlying node(s).
Timing is so important to the StorageGRID functionality that the NTP service is a dependency for all other grid services; the servermanager will not bring any other services online if NTP does not start successfully. In fact, this is utilized in the maintenance procedures. It is possible to prevent an auto-startup of the StorageGRID software by creating a 0-byte file: /etc/sv/ntp/DoNotStart
A best practice for installing StorageGRID is to configure NTP in a hierarchical manner - the Control Nodes are configured for time synchronization with external time sources. They are also configured as synchronizing peers (that is, the Control Nodes synchronize with each other as well as the external sources). All other nodes within the StorageGRID use Control Nodes as their time sources.
If a sufficient number of Control Nodes deviate from the external time sources, the entire grid might begin to drift away from the external time sources.
Dramatic time drifts from the configured sources are referred to as 'fly-wheeling', and can affect either single nodes or the whole grid.
To ensure the best possible conditions for StorageGRID to maintain time precision and accuracy, NTP configuration should adhere to the following best practices:
- Use Stratum 1 time sources. Stratum 0 are generally atomic or radio clocks that are not accessible through the network. Stratum 1 sources are servers directly connected to the Stratum 0 sources and are therefore the most accurate NTP sources available. A complete description of NTP strata is available on the Wikipedia NTP page.
- Use between 4 and 8 Stratum 1 sources. Using multiple sources increases the probability of an NTP lock in the event of upstream network disruptions, but too many sources adds complexity without benefit. A large number of sources can complicate remediation efforts if the node or grid loses NTP lock and begins flywheeling.
- Choose geographically local sources when selecting. This will reduce both the network latency and the likelihood of network disruption to a given source.
- Always test the connectivity to a new time source before incorporating it into the NTP configuration.
- Use these resources to locate additional NTP sources:
- For more information, see Selecting Offsite NTP Servers.
At the time of writing, the following NTP Stratum 1 sources are commonly used in StorageGRID deployments:
- Pacific Time Zone
tick.ucla.edu (164.67.62.194)
utcnist.colorado.edu (128.138.140.44)
timekeeper.isi.edu (128.9.176.30)
usno.pa-x.dec.com (204.123.2.72)
- Mountain Time Zone
utcnist.colorado.edu (128.138.140.44)
utcnist2.colorado.edu (128.138.141.172)
usno.pa-x.dec.com (204.123.2.72)
- Central Time Zone
time.nist.gov (192.43.244.18)
nist1-lnk.binary.net (216.229.0.179)
nist.time.nosc.us (96.226.123.117)
- Eastern Time Zone
navobs1.gatech.edu (130.207.244.240)
nist1-ny.ustiming.org (64.90.182.55)
nist1-nj.ustiming.org (96.47.67.105)
Additional Information
additionalInformation_text