RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1443342 - chronyd rejects PTP_SYS_OFFSET readings with zero delay
Summary: chronyd rejects PTP_SYS_OFFSET readings with zero delay
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: chrony
Version: 7.4
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: ---
Assignee: Miroslav Lichvar
QA Contact: HuijingHei
URL:
Whiteboard:
Depends On:
Blocks: 1388419
TreeView+ depends on / blocked
 
Reported: 2017-04-19 06:27 UTC by HuijingHei
Modified: 2017-08-01 16:20 UTC (History)
10 users (show)

Fixed In Version: chrony-3.1-2.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-08-01 16:20:39 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:1908 0 normal SHIPPED_LIVE chrony bug fix and enhancement update 2017-08-01 17:54:45 UTC

Description HuijingHei 2017-04-19 06:27:36 UTC 
Description of problem:
When set time server with /dev/ptp0, chronyd runs with failed logs "sys_linux.c:749:(get_precise_phc_sample) ioctl(PTP_SYS_OFFSET_PRECISE) failed : Invalid argument"

Version-Release number of selected component (if applicable):
Kernel: 3.10.0-654.el7.x86_64
Chrony: 3.1-1.el7.x86_64
Host:   Hyper-V Windows Server 2016, Gen2


How reproducible: 100%

Steps to Reproduce:
1. Make sure "Time synchronization" is enabled, start vm
2. Check that /dev/ptp0 is present
# ls /dev/ptp0

3. Update /etc/chrony.conf to comment all servers and add /dev/ptp0 clock source.
1) Comment out all "server ..." lines, add "#" in the beginning of the line
2) Add PTP source:
       refclock PHC /dev/ptp0 poll 3 dpoll -2 offset 0 trust
3) Comment "makestep 10 3" line, add "makestep 1 -1", to allow the step on any clock update

4. Ignore any NTP server information provided by DHCP by setting PEERNTP=no:
Add "PEERNTP=no" to /etc/sysconfig/network-scripts/ifcfg-eth0
5. Delete file /var/lib/dhclient/chrony.servers.eth0
# rm -f /var/lib/dhclient/chrony.servers.eth0

6. Reboot vm
7. Stop chronyd, run chronyd manually in verbose mode, and run 'chronyc sources' in another console
# systemctl stop chronyd
# chronyd -d -d
# chronyc sources

Actual results:
Chronyd runs with failed logs

# chronyd -d -d
2017-04-19T02:08:30Z main.c:473:(main) chronyd version 3.1 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SECHASH +SIGND +ASYNCDNS +IPV6 +DEBUG)
2017-04-19T02:08:30Z conf.c:406:(CNF_ReadFile) Reading /etc/chrony.conf
2017-04-19T02:08:30Z local.c:149:(calculate_sys_precision) Clock precision 0.000000100 (-23)
2017-04-19T02:08:30Z sys_linux.c:317:(get_version_specific_details) Linux kernel major=3 minor=10 patch=0
2017-04-19T02:08:30Z sys_linux.c:338:(get_version_specific_details) hz=100 nominal_tick=10000 max_tick_bias=1000
2017-04-19T02:08:30Z local.c:663:(lcl_RegisterSystemDrivers) Local freq=-8.490ppm
2017-04-19T02:08:30Z refclock.c:274:(RCL_AddRefclock) refclock PHC refid=PHC0 poll=3 dpoll=-2 filter=32
2017-04-19T02:08:30Z util.c:1172:(UTI_DropRoot) Dropped root privileges: UID 993 GID 989
2017-04-19T02:08:30Z reference.c:209:(REF_Initialise) Frequency -8.490 +/- 0.015 ppm read from /var/lib/chrony/drift
2017-04-19T02:08:30Z sys_generic.c:251:(update_slew) slew offset=0.000000e+00 corr_rate=0.000000e+00 base_freq=-8.489740 total_freq=-8.489731 slew_freq=9.164961e-12 duration=10000.000000 slew_error=1.008146e-13
2017-04-19T02:08:31Z sys_linux.c:749:(get_precise_phc_sample) ioctl(PTP_SYS_OFFSET_PRECISE) failed : Invalid argument
2017-04-19T02:08:31Z sys_linux.c:749:(get_precise_phc_sample) ioctl(PTP_SYS_OFFSET_PRECISE) failed : Invalid argument
2017-04-19T02:08:31Z sys_linux.c:749:(get_precise_phc_sample) ioctl(PTP_SYS_OFFSET_PRECISE) failed : Invalid argument
2017-04-19T02:08:31Z sys_linux.c:749:(get_precise_phc_sample) ioctl(PTP_SYS_OFFSET_PRECISE) failed : Invalid argument
2017-04-19T02:08:32Z sys_linux.c:749:(get_precise_phc_sample) ioctl(PTP_SYS_OFFSET_PRECISE) failed : Invalid argument
2017-04-19T02:08:32Z cmdmon.c:1304:(read_from_cmd_socket) Received 32 bytes from /var/run/chrony/chronyc.13824.sock fd 7
2017-04-19T02:08:32Z cmdmon.c:405:(transmit_reply) Sent 32 bytes to /var/run/chrony/chronyc.13824.sock fd 7
2017-04-19T02:08:32Z cmdmon.c:1304:(read_from_cmd_socket) Received 76 bytes from /var/run/chrony/chronyc.13824.sock fd 7
2017-04-19T02:08:32Z cmdmon.c:405:(transmit_reply) Sent 76 bytes to /var/run/chrony/chronyc.13824.sock fd 7
2017-04-19T02:08:33Z sys_linux.c:749:(get_precise_phc_sample) ioctl(PTP_SYS_OFFSET_PRECISE) failed : Invalid argument
2017-04-19T02:08:33Z sys_linux.c:749:(get_precise_phc_sample) ioctl(PTP_SYS_OFFSET_PRECISE) failed : Invalid argument
2017-04-19T02:08:33Z sys_linux.c:749:(get_precise_phc_sample) ioctl(PTP_SYS_OFFSET_PRECISE) failed : Invalid argument
2017-04-19T02:08:33Z sys_linux.c:749:(get_precise_phc_sample) ioctl(PTP_SYS_OFFSET_PRECISE) failed : Invalid argument
2017-04-19T02:08:34Z cmdmon.c:1304:(read_from_cmd_socket) Received 32 bytes from /var/run/chrony/chronyc.13825.sock fd 7
2017-04-19T02:08:34Z cmdmon.c:405:(transmit_reply) Sent 32 bytes to /var/run/chrony/chronyc.13825.sock fd 7
2017-04-19T02:08:34Z cmdmon.c:1304:(read_from_cmd_socket) Received 76 bytes from /var/run/chrony/chronyc.13825.sock fd 7
2017-04-19T02:08:34Z cmdmon.c:405:(transmit_reply) Sent 76 bytes to /var/run/chrony/chronyc.13825.sock fd 7
2017-04-19T02:08:35Z sys_linux.c:749:(get_precise_phc_sample) ioctl(PTP_SYS_OFFSET_PRECISE) failed : Invalid argument

# chronyc sources
210 Number of sources = 1
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
#? PHC0                          0   1     0     -     +0ns[   +0ns] +/-    0ns


Expected results:
chronyd is running normally

Additional info:
Have tried chrony-2.1.1-3.el7.x86_64 with Kernel-3.10.0-654.el7.x86_64, chrony works normally.
Comment 2 Miroslav Lichvar 2017-04-19 09:13:45 UTC 
It seems there are two separate issues.

One is that the PTP_SYS_OFFSET_PRECISE ioctl doesn't work. From the kernel code it looks like it should be supported. I'm not sure where the EINVAL is coming from, but the discussion probably belongs to the kernel bug #1388419.

The second issue is that chronyd rejects data from the PTP_SYS_OFFSET ioctl when there is a negative or zero difference between two consecutive system times. With real PTP clocks that is not expected to happen as it normally takes at least a microsecond to read the PTP clock over PCIe bus, but with a low-resolution system clock and a fast virtual clock, it's apparently possible to get two readings of the same system time, which chronyd rejects as invalid.

chrony versions before 3.1 didn't have this check, so this is a regression. Please note that this can happen only with the PTP_SYS_OFFSET ioctl, which is used when the PTP_SYS_OFFSET_PRECISE ioctl failed, or the PHC refclock was configured with the nocrossts option.

The fix is to modify the check to allow zero and reject only readings with negative delay.
Comment 4 Miroslav Lichvar 2017-04-19 11:20:10 UTC 
Upstream fix:

https://git.tuxfamily.org/chrony/chrony.git/commit/?id=abb09418b16993ecd8289dd459dff91701f4f971
Comment 5 Miroslav Lichvar 2017-04-24 15:08:01 UTC 
Huijing, can you please test chrony-3.1-2.el7 if it can synchronize to the PHC? 

There should be only one "ioctl(PTP_SYS_OFFSET_PRECISE) failed : Invalid argument" error message and then it should switch to the PTP_SYS_OFFSET ioctl.

If PTP_SYS_OFFSET_PRECISE is working, chronyd can be forced to use PTP_SYS_OFFSET with the nocrossts option, e.g. change the line in chrony.conf to

refclock PHC /dev/ptp0:nocrossts poll 3 dpoll -2 offset 0 trust
Comment 7 HuijingHei 2017-04-25 10:13:05 UTC 
(In reply to Miroslav Lichvar from comment #5)
> Huijing, can you please test chrony-3.1-2.el7 if it can synchronize to the
> PHC? 
> 
> There should be only one "ioctl(PTP_SYS_OFFSET_PRECISE) failed : Invalid
> argument" error message and then it should switch to the PTP_SYS_OFFSET
> ioctl.
> 
> If PTP_SYS_OFFSET_PRECISE is working, chronyd can be forced to use
> PTP_SYS_OFFSET with the nocrossts option, e.g. change the line in
> chrony.conf to
> 
> refclock PHC /dev/ptp0:nocrossts poll 3 dpoll -2 offset 0 trust


Hi Miroslav,

Have tested chrony-3.1-2.el7 in RHEL7.4 on Hyperv 2016, and it can synchronize to the PHC.

The results are the same as your comment. Change the bug status to Verified.

Thanks
hhei
Comment 8 errata-xmlrpc 2017-08-01 16:20:39 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1908
Note You need to log in before you can comment on or make changes to this bug.