Last time, I built a stratum 1 NTP server with a PPS signal from a GPS receiver, synchronizing my server’s clock to within 10 microseconds of UTC. However, NTP was designed to synchronize clocks within a few tens of milliseconds over the Internet, and I’d be lucky to achieve millisecond accuracy on a LAN. I mentioned that PTP was the alternative that could achieve accuracy in the sub-microsecond range. Well, this time I’ll be setting up PTP between my server and my PC with the hardware timestamping on the ConnectX-3s.

If you are following along at home, don’t despair if your hardware can’t do timestamping or PTP. I will also attempt to set up PTP with software timestamping later for my other devices.

Naturally, I first turned to the gpsd documentation, since that was a decent reference for setting up NTP with the PPS signal. Well, this is what it says for PTP with hardware timestamping:

Sadly, theory and practice diverge here. I have never succeeded in making hardware timestamping work. I have successfully trashed my host system clock. Tread carefully. If you make progress please pass on some clue.

That didn’t sound encouraging at all. “Oh well, I guess I am on my own here,” I thought to myself. “How bad could digging through a few man pages and random online documentation be? Worst case, there is the source code, right?”

Installing linuxptp

The PTP implementation that everyone talks about is linuxptp, so I started by installing it on my server:

$ sudo apt install linuxptp
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  linuxptp
0 upgraded, 1 newly installed, 0 to remove and 34 not upgraded.
Need to get 177 kB of archives.
After this operation, 773 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian bullseye/main amd64 linuxptp amd64 3.1-2.1 [177 kB]
Fetched 177 kB in 0s (6,335 kB/s)
Selecting previously unselected package linuxptp.
(Reading database ... 506647 files and directories currently installed.)
Preparing to unpack .../linuxptp_3.1-2.1_amd64.deb ...
Unpacking linuxptp (3.1-2.1) ...
Setting up linuxptp (3.1-2.1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/timemaster.service → /lib/systemd/system/timemaster.service.
Processing triggers for man-db (2.9.4-2) ...

It installed this timemaster service, which is a configuration generator to make it possible to run NTP with PTP as a clock source. It seems like an interesting tool, but my server would be the clock source for the entire network (“grandmaster” in PTP terminology), so I definitely wouldn’t want it. It doesn’t help that the default configuration file that comes with the linuxptp package doesn’t seem to work and the Debian documentation is non-existent1, so let’s just disable it:

$ sudo systemctl disable --now timemaster.service
Removed /etc/systemd/system/multi-user.target.wants/timemaster.service.

Now, the actual underlying linuxptp utilities are:

  • ptp4l, which is the actual PTP daemon. It announces “PTP time” (really TAI) in hardware timestamping mode and UTC in software timestamping mode; and
  • phc2sys, which synchronizes between the PTP hardware clock (PHC) and the system clock. When running ptp4l in hardware timestamping mode, the hardware clock needs to be synchronized to the system clock.

Setting up hardware timestamping PTP

I decided to dive straight into the exciting bit, because why not? I started by running ptp4l between my server and PC on the ConnectX-3s.

Now, there are three possible transports for PTP:

  • IEEE 802.3 network transport, which is a fancy way of saying raw Ethernet with EtherType 0x88F7. This is option -2 in ptp4l;
  • UDP IPv4 network transport. This is option -4 in ptp4l; and
  • UDP IPv6 network transport. This is option -6 in ptp4l.

Note that for UDP transport, this typically is done over multicast.

I ended up choosing the IEEE 802.3 transport since I am using Linux bridges to link my ConnectX-3 to the rest of my Ethernet network and bridges don’t support hardware timestamping. I didn’t feel like fiddling with multicast on an interface that has no IP, so raw Ethernet it was. In a less janky setup with a real 40 GbE switch, I’d probably have used multicast UDP.

On the server, I ran (pretend cx3 is the name for the ConnectX-3 interface, and -m is for printing to stdout instead of syslog):

$ sudo ptp4l -2mi cx3
ptp4l[94881.798]: selected /dev/ptp0 as PTP clock
ptp4l[94881.858]: port 1: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[94881.858]: port 0: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[94888.005]: port 1: LISTENING to MASTER on ANNOUNCE_RECEIPT_TIMEOUT_EXPIRES
ptp4l[94888.005]: selected local clock [server EUI-64] as best master
ptp4l[94888.005]: port 1: assuming the grand master role

On the client, I ran (-s to mean a “slave” clock):

$ sudo ptp4l -2msi cx3
ptp4l[17633.604]: selected /dev/ptp1 as PTP clock
ptp4l[17633.650]: port 1: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[17633.650]: port 0: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[17634.466]: port 1: new foreign master [server EUI-64]-1
ptp4l[17638.466]: selected best master clock [server EUI-64]
ptp4l[17638.466]: port 1: LISTENING to UNCALIBRATED on RS_SLAVE
ptp4l[17640.466]: master offset      11088 s0 freq  -20470 path delay       138
ptp4l[17641.466]: master offset      11286 s2 freq  -20272 path delay       155
ptp4l[17641.466]: port 1: UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED
ptp4l[17642.466]: master offset      11300 s2 freq   -8972 path delay       155
ptp4l[17643.466]: master offset         29 s2 freq  -16853 path delay       155
ptp4l[17644.466]: master offset      -3360 s2 freq  -20233 path delay       155
ptp4l[17645.466]: master offset      -3388 s2 freq  -21269 path delay       191
ptp4l[17646.466]: master offset      -2317 s2 freq  -21215 path delay       156
ptp4l[17647.466]: master offset      -1362 s2 freq  -20955 path delay       191
ptp4l[17648.466]: master offset       -640 s2 freq  -20641 path delay       191
ptp4l[17649.466]: master offset       -203 s2 freq  -20396 path delay       191
...
ptp4l[17660.466]: master offset         -3 s2 freq  -20190 path delay       234
ptp4l[17661.466]: master offset         -3 s2 freq  -20191 path delay       234
ptp4l[17662.466]: master offset          2 s2 freq  -20187 path delay       235
...

That actually just worked. The ConnectX-3 NICs were synchronized to within a few nanoseconds of each other after less than half a minute.

Now, I need to feed the correct time into the server’s PTP hardware clock with phc2sys (-w to wait for ptp4l to be ready, -m to print to stdout instead of syslog, -c is the network interface):

$ sudo phc2sys -s CLOCK_REALTIME -c cx3 -wm
phc2sys[95992.748]: /dev/ptp0 sys offset -20572648 s0 freq +218028 delay   4440
phc2sys[95993.748]: /dev/ptp0 sys offset -20572410 s1 freq +218266 delay   4440
phc2sys[95994.748]: /dev/ptp0 sys offset        -4 s2 freq +218262 delay   4450
phc2sys[95995.749]: /dev/ptp0 sys offset         1 s2 freq +218266 delay   4440
phc2sys[95996.749]: /dev/ptp0 sys offset         5 s2 freq +218270 delay   4450

Note that the -d option is supposed to allow you to pass a PPS device, but apparently, that’s only for disciplining CLOCK_REALTIME with a PPS device from the NIC. Therefore, I can’t use the PPS signal from my GPS directly. Instead, I have to rely on NTP’s adjustments to CLOCK_REALTIME.

Now on the client, I’ll need to disable NTP to avoid it interfering with the PTP signal. By default, Debian uses systemd-timesyncd, which can be disabled via sudo timedatectl set-ntp false. Otherwise, try either sudo systemctl disable --now ntpd.service or sudo systemctl disable --now chronyd.service.

Now, I should be able to sync the client’s clock to it (-c CLOCK_REALTIME is implied, -s is the network interface):

$ sudo phc2sys -s /dev/ptp1 -wm
phc2sys[18917.156]: CLOCK_REALTIME phc offset -19059065 s0 freq  -11778 delay   4381
phc2sys[18918.157]: CLOCK_REALTIME phc offset -19080139 s1 freq  -32848 delay   4450
phc2sys[18919.157]: CLOCK_REALTIME phc offset     -8422 s2 freq  -41270 delay   4440
...
phc2sys[18929.158]: CLOCK_REALTIME phc offset        42 s2 freq  -41245 delay   4420
phc2sys[18930.158]: CLOCK_REALTIME phc offset        -2 s2 freq  -41277 delay   4431
phc2sys[18931.158]: CLOCK_REALTIME phc offset        43 s2 freq  -41232 delay   4450
phc2sys[18932.159]: CLOCK_REALTIME phc offset       -35 s2 freq  -41297 delay   4410
...

Alright, it seemed to work! As you can see, it corrected a “massive” 19 ms error on my PC’s clock (thanks, systemd-timesyncd2) and moved it within some nanoseconds of my server, whose clock is at least decently accurate due to the PPS signal from the GPS. I checked the time on my system for sanity and it appeared accurate, so I didn’t end up trashing my system clock.

There is a bit of jitter, so I am only comfortable with saying that this achieved sub-microsecond accuracy to the server, which is still pretty good. The server’s real-time clock should be accurate to within a few microseconds of UTC due to the PPS signal. Now I just need to daemonize this.

(Note: If you decide to stop here, remember to re-enable NTP.)

Automatically starting ptp4l and phc2sys

Now that I have ptp4l and phc2sys working, I should make them services that start automatically and use proper configuration files too.

For the server, I decided to use the following ptp4l configuration file (at /etc/linuxptp/ptp4l.conf, remember to replace cx3 with your interface name):

[global]
# Only syslog every 1024 seconds
summary_interval 10

# Increase priority to allow this server to be chosen as the PTP grandmaster.
priority1 10
priority2 10

[cx3]
network_transport L2

For the client, I decided to the following configuration file:

[global]
# Only syslog every 1024 seconds
summary_interval 10

[cx3]
network_transport L2

And the following systemd unit (at /etc/systemd/system/ptp4l.service) on both:

[Unit]
Description=Precision Time Protocol service
Documentation=man:ptp4l

[Service]
Type=simple
ExecStart=/usr/sbin/ptp4l -f /etc/linuxptp/ptp4l.conf

[Install]
WantedBy=multi-user.target

Now just run sudo systemctl enable --now ptp4l.service to start ptp4l on both machines.

For phc2sys, the configuration file is pointless since we can’t put the interface names into it. I decided to instead configure it with the systemd unit. On the server, I created /etc/systemd/system/phc2sys.service (replace cx3 with your interface name, -u 1024 means printing a summary every 1024 seconds to avoid log spam):

[Unit]
Description=Synchronizing PTP clock to system time
Documentation=man:phc2sys
After=ptp4l.service

[Service]
Type=simple
ExecStart=/usr/sbin/phc2sys -s CLOCK_REALTIME -c cx3 -w -u 1024

[Install]
WantedBy=multi-user.target

On the client, I used the following systemd unit:

[Unit]
Description=Synchronizing system time to PTP
Documentation=man:phc2sys
After=ptp4l.service

[Service]
Type=simple
ExecStart=/usr/sbin/phc2sys -s cx3 -w -u 1024

[Install]
WantedBy=multi-user.target

Now just run sudo systemctl enable --now phc2sys.service to start phc2sys on both machines. Now my PC should be synchronized as much as possible to the server, which is synchronized to GPS time.

Software PTP

For fun, I decided to run PTP client on the Raspberry Pi 3 to see how well it syncs itself to my server. (This would also be rather indicative of what would happen if I tried to get a PPS time signal out of the Raspberry Pi 3.)

Unfortunately, ptp4l doesn’t interact well with bridge interfaces or 802.3ad bonds, so for this testing, I removed an interface from the 802.3ad link aggregation I have to my switch.

To avoid interference with my existing setup, I am using PTP domain 1 for this test (you don’t need --domain=1 if it’s the only ptp4l instance on your LAN). I ran the following command on the server (where rtl8111 is the interface name):

$ sudo ptp4l -2Smi rtl8111 --domain=1
ptp4l[170800.930]: port 1: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[170800.930]: port 0: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[170808.503]: port 1: LISTENING to MASTER on ANNOUNCE_RECEIPT_TIMEOUT_EXPIRES
ptp4l[170808.503]: selected local clock [server EUI-64] as best master
ptp4l[170808.503]: port 1: assuming the grand master role

Now on the Raspberry Pi, I ran:

$ sudo ptp4l -2Ssmi eth0 --domain=1
ptp4l[619306.404]: port 1: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[619306.406]: port 0: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[619307.984]: port 1: new foreign master [server EUI-64]-1
ptp4l[619311.984]: selected best master clock [server EUI-64]
ptp4l[619311.985]: foreign master not using PTP timescale
ptp4l[619311.985]: port 1: LISTENING to UNCALIBRATED on RS_SLAVE
ptp4l[619313.997]: master offset     832985 s0 freq   -2743 path delay    177882
...
ptp4l[619329.998]: master offset     863399 s1 freq    -842 path delay    177707
ptp4l[619330.998]: master offset      -8896 s2 freq   -1741 path delay    177707
ptp4l[619330.998]: port 1: UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED
ptp4l[619331.998]: master offset      58508 s2 freq   +5058 path delay    177707
ptp4l[619332.998]: master offset      -7400 s2 freq   -1540 path delay    178331
...
ptp4l[619429.004]: master offset       6511 s2 freq     -35 path delay    174225
ptp4l[619430.004]: master offset       4366 s2 freq    -245 path delay    174225
ptp4l[619431.004]: master offset      50330 s2 freq   +4402 path delay    174225
ptp4l[619432.004]: master offset      -6701 s2 freq   -1308 path delay    181328
ptp4l[619433.004]: master offset      -1725 s2 freq    -812 path delay    182561
ptp4l[619434.004]: master offset      -9388 s2 freq   -1588 path delay    182561
ptp4l[619435.004]: master offset     -14759 s2 freq   -2140 path delay    182873
ptp4l[619436.004]: master offset     -12987 s2 freq   -1975 path delay    183020
ptp4l[619437.004]: master offset       8435 s2 freq    +175 path delay    183020
ptp4l[619438.004]: master offset      -7114 s2 freq   -1387 path delay    183020

Note that in software timestamping mode, ptp4l directly messes with the system clock, and no phc2sys is needed.

As we can see, ptp4l struggled to make the offset fall under 10 μs due to the software timestamping and the USB jitter. If I had used the Raspberry Pi 3 to receive the PPS signal, this would be the best accuracy I could possibly achieve.

Now, I also have an Atomic Pi with a PCIe NIC (RTL8111), though it also lacks hardware timestamping. Let’s see how it fares:

$ sudo ptp4l -2Ssmi enp1s0 --domain=1
ptp4l[574.866]: port 1: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[574.866]: port 0: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[574.993]: port 1: new foreign master [server EUI-64]-1
ptp4l[578.993]: selected best master clock [server EUI-64]
ptp4l[578.994]: foreign master not using PTP timescale
ptp4l[578.994]: port 1: LISTENING to UNCALIBRATED on RS_SLAVE
ptp4l[580.031]: master offset    7185769 s0 freq  +28159 path delay     38385
...
ptp4l[596.032]: master offset    7608532 s1 freq  +54579 path delay     40711
ptp4l[597.032]: master offset      -7046 s2 freq  +53868 path delay     40711
ptp4l[597.032]: port 1: UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED
ptp4l[598.032]: master offset      -1398 s2 freq  +54431 path delay     40711
ptp4l[599.032]: master offset      -2483 s2 freq  +54320 path delay     40985
ptp4l[600.032]: master offset       1492 s2 freq  +54719 path delay     40985
ptp4l[601.032]: master offset       4112 s2 freq  +54985 path delay     42100
ptp4l[602.032]: master offset       3030 s2 freq  +54880 path delay     42100
ptp4l[603.032]: master offset      -8799 s2 freq  +53688 path delay     42983
ptp4l[604.032]: master offset        400 s2 freq  +54609 path delay     42983
ptp4l[605.032]: master offset       -119 s2 freq  +54557 path delay     43305
ptp4l[606.032]: master offset      -3942 s2 freq  +54170 path delay     42853
ptp4l[607.032]: master offset       2211 s2 freq  +54788 path delay     42403
ptp4l[608.032]: master offset       2455 s2 freq  +54815 path delay     42853
ptp4l[609.032]: master offset       -606 s2 freq  +54508 path delay     43222
ptp4l[610.032]: master offset        490 s2 freq  +54618 path delay     43222
ptp4l[611.032]: master offset      -6645 s2 freq  +53898 path delay     42237
ptp4l[612.032]: master offset       4334 s2 freq  +55000 path delay     42089
ptp4l[613.032]: master offset       1950 s2 freq  +54764 path delay     42089
...

As you can see, it’s struggling less hard than the Raspberry Pi, but can only keep the clock within a few microseconds. This is fairly decent if you don’t have PTP-capable hardware.

Given that PTP struggles to work with 802.3ad link aggregation, it means that I’d have to have a NIC dedicated to serving PTP on the server. This doesn’t seem reasonable, so I won’t be deploying PTP for hosts other than my PC. As such, converting the commands shown here into a configuration file for systemd is left as an exercise for the reader.

Conclusion

With hardware timestamping NICs at both ends, it’s easy to achieve sub-microsecond accuracy with PTP.

With software timestamping NICs at both ends, it’s possible to achieve sub-10 μs accuracy with PTP.

With a Raspberry Pi, it’s possible to achieve sub-100 μs accuracy with PTP.

I hope this proved a useful and enjoyable read.

Notes

  1. The README.Debian file has the following to say about timemaster:

    The service timemaster also isn’t enabled and started by default [sic]

    That’s it. Not even a period to end the last sentence in the file. And the service is enabled by default even though it won’t start due to lack of chronyd in the default Debian install. It also mentions systemd services ptp4l and phc2sys that don’t exist. ↩

  2. To be fair, systemd-timesyncd is designed to be lightweight and never designed to be more accurate than tens of milliseconds. It’s not like this level of precision is needed for normal desktop use. ↩