Description of problem: The kickstart snippet for ensuring a network time synchronization service (ntpd or chrony) is installed and enabled is missing for VMs. Time is intermittently going backwards on the Guests without this configuration. Version-Release number of selected component (if applicable): Beaker 19.0 How reproducible: Intermittently Actual results: https://beaker.engineering.redhat.com/jobs/826571 https://beaker.engineering.redhat.com/recipes/1717222 http://beaker-archive.app.eng.bos.redhat.com/beaker-logs/2014/12/8265/826571/1717222/console.log -----<-snip->----- logger: /usr/bin/rhts-test-runner.sh rhts-test-update 127.0.0.1:7086 26683607 start kernel-kernel-standards-usex-1.9-29 2014-12-15 01:45:30,956 rhts_task checkin_start: INFO setting nohup 2014-12-15 01:45:30,998 backend _send_cmd: INFO Command ['Command', 'forward', '5728139d-3e00-4c16-bbaf-43e964eb81d8', {'event': ['Event', 'extend_watchdog', '0b56cba8-2165-4d16-8177-484500fbae59', {'source': 'beah_beaker_backend', 'id': '3c4d4745-a7d5-4176-9a13-e4a34e61c16c'}, None, {'timeout': 3594}]}] sent. 12/15/14 01:45:30 testID:26683607 start: /mnt/tests/kernel/standards/usex/1.9-29 /mnt/tests/kernel/standards/usex/1.9-29 logger: 2014-12-15 01:45:56 /usr/bin/rhts-test-runner.sh 5554 3780 hearbeat... logger: 2014-12-14 20:46:57 /usr/bin/rhts-test-runner.sh 5554 3840 hearbeat... logger: 2014-12-14 20:47:57 /usr/bin/rhts-test-runner.sh 5554 3900 hearbeat... -----<-snip->----- Expected results: Guest time syncs are correct. Additional info:
All, We see this was an intentional change in Beaker 19: https://beaker-project.org/docs/whats-new/release-19.html#network-time-syncing-is-disabled-for-vms However, This is a regression and causes issues during Kernel Tier1 testing. Best, -pbunyan
some additional info.. https://s19n.net/articles/2011/kvm_clock.html That document pointed out that our own documentation recommends that you run ntp on the guest. https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Virtualization_for_Servers/2.2/html/Administration_Guide/chap-Virtualization-KVM_guest_timing_management.html "The Network Time Protocol (NTP) daemon should be running on the host and the virtual machines. "
another link http://thread.gmane.org/gmane.comp.emulators.kvm.devel/96835/focus=96981 It looks to me like we should continue to run ntp on the guests but configure them to use the physical host. That means the physical host has to be configured to let the guests sync from it right?
(In reply to Bill Peck from comment #2) > https://access.redhat.com/documentation/en-US/ > Red_Hat_Enterprise_Virtualization_for_Servers/2.2/html/Administration_Guide/ > chap-Virtualization-KVM_guest_timing_management.html > > "The Network Time Protocol (NTP) daemon should be running on the host and > the virtual machines. " Hmm, I'm sure I have seen completely contradictory documentation somewhere in the past... not sure where though. Are the kernel tier 1 tests running guests for so long that the guest clock falls badly out of sync? KVM does at least guarantee us that the guest has an accurate clock when it is first started, right? The actual reason for disabling clock syncing in guests was for OpenStack support. In the OpenStack environment we were using, there is no NTP servers specified in DHCP. But I guess that is something we need to get fixed in that environment. So we can revert the change in 19.2. As a workaround in the meantime, you can add !no_clock_sync to the guest ksmeta.
(In reply to Dan Callaghan from comment #4) > Are the kernel tier 1 tests running guests for so long that the guest clock > falls badly out of sync? Problem was test (which relies on system time) failing to complete, because time jumped 5 hours to past, so it hit LWD. We have at least 2 tests that stop/start ntpd / call ntpdate. > KVM does at least guarantee us that the guest has > an accurate clock when it is first started, right? I think problem on RHEL6 is that initial time is in UTC, but /etc/adjtime says that RTC is LOCAL time. + It is ~11.41 CET (GMT+1) + kernel boots | rtc_cmos 00:01: setting system clock to 2014-12-16 10:41:32 UTC + timezone is set to "America/New_York" + # date | Tue Dec 16 05:42:43 EST 2014 + # cat /etc/adjtime | 0.000000 0.0 0.000000 | 0.0 | LOCAL + udev calls "/sbin/hwclock --systz --rtc=/dev/rtc0" + # date | Tue Dec 16 10:43:26 EST 2014 --> PROBLEM, time is now wrong, and presumably will stay wrong, unless ntpdate/ntpd runs This is from 2 machines in BOS lab, both have "timezone America/New_York" in kickstart: RHEL-6.6-20140926.0 # cat /etc/adjtime 0.000000 0.000000 0.000000 0.000000 LOCAL RHEL-7.1-20141204.2 # cat /etc/adjtime 0.0 0 0.0 0 UTC I think we haven't seen this on RHEL7 so far. Adding Miroslav to CC.
Yes, the adjtime UTC/LOCAL setting needs to match the kvm setting (by default UTC) in order to have correct time after boot. If the guest's system clock should keep a sub-second accuracy, it should also run an NTP client. As an NTP server it can use the host, but it shouldn't be necessary unless trying to get the best quality of timekeeping.
Hint: I didn't test this, but --utc parameter for timezone kickstart option might be one way to fix /etc/adjtime to contain "UTC". BTW, Has this "no NTP" change been applied only to dynamically created guests or also static guests with "KVM/PowerKVM" hypervisor?
(In reply to Jan Stancek from comment #7) > BTW, Has this "no NTP" change been applied only to dynamically created > guests or also static guests with "KVM/PowerKVM" hypervisor? No, only dynamically created ones. The assumption was that they should have accurate clock when they are created. So if we were previously relying on ntpd to fix some mismatch in UTC/localtime for the RTC then maybe it is better to fix that problem properly?
(In reply to Jan Stancek from comment #7) > Hint: I didn't test this, but --utc parameter for timezone kickstart option > might be one way to fix /etc/adjtime to contain "UTC". This sounds like the right answer to me. The libvirt default is to put the guest clock in UTC, and /distribution/virt/install doesn't ever change it. If you want to try this out in kernel testing right away, you can hack it into timezone ksmeta like this: ks_meta="timezone='--utc America/New_York'"
(In reply to Dan Callaghan from comment #8) > So if we were previously relying on ntpd to fix some mismatch in > UTC/localtime for the RTC then maybe it is better to fix that problem > properly? I think we should do both, but also restore old behaviour. It worked for years, there can be users who rely on accurate clock for their testing on guests. 1. fix UTC/LOCAL mismatch for dynamically created guests This will need to be tested, to see if it works as we think and to see on what major releases it is supported. 2. start NTP/chrony by default everywhere (as we did in past) 3. provide ks_meta option which allows to turn it off (on any system, host or guest), openstack can use it if that's what's needed
(In reply to Dan Callaghan from comment #9) > If you want to try this out in kernel testing right away, you can hack it > into timezone ksmeta like this: > > ks_meta="timezone='--utc America/New_York'" This seems to work as expected (J:828519). On RHEL-6.6-20140926.0 Server: "timezone --utc America/New_York" is in kickstart and after boot I see: # cat /etc/adjtime 0.0 0 0.0 0 UTC
(In reply to Jan Stancek from comment #10) > 3. provide ks_meta option which allows to turn it off (on any system, host > or guest), openstack can use it if that's what's needed Yeah we have that already, it's no_clock_sync. > 2. start NTP/chrony by default everywhere (as we did in past) Matt's patch here does this (unsetting no_clock_sync): http://gerrit.beaker-project.org/3875 > 1. fix UTC/LOCAL mismatch for dynamically created guests > This will need to be tested, to see if it works as we think and to see on > what major releases it is supported. Matt, can you upload a separate patch for this part? I think we just need --utc in the timezone command for guest kickstarts.
(In reply to Jan Stancek from comment #10) > 1. fix UTC/LOCAL mismatch for dynamically created guests > This will need to be tested, to see if it works as we think and to see on > what major releases it is supported. I couldn't find any kickstart docs for RHEL3 or RHEL4 but it looks like timezone command has understood --utc since at least 1999 [1] so I think we are safe on that front :-) But yes we will need to run a full matrix to check that it has the expected effect. [1] https://git.fedorahosted.org/cgit/anaconda.git/commit/?id=23474f62a459bcb70ab814f67279409e028a28e2
(In reply to Dan Callaghan from comment #12) > (In reply to Jan Stancek from comment #10) > > 3. provide ks_meta option which allows to turn it off (on any system, host > > or guest), openstack can use it if that's what's needed > > Yeah we have that already, it's no_clock_sync. > > > 2. start NTP/chrony by default everywhere (as we did in past) > > Matt's patch here does this (unsetting no_clock_sync): > http://gerrit.beaker-project.org/3875 > > > 1. fix UTC/LOCAL mismatch for dynamically created guests > > This will need to be tested, to see if it works as we think and to see on > > what major releases it is supported. > > Matt, can you upload a separate patch for this part? I think we just need > --utc in the timezone command for guest kickstarts. On Gerrit, http://gerrit.beaker-project.org/#/c/3882/ Cheers, Matt Jia
Suggestions for verification: Run a matrix of RHEL5, RHEL6, RHEL7 host x RHEL5, RHEL6, RHEL7 guest. In the guest recipes set ks_meta="no_clock_sync", and run a command like this: <task name="/distribution/command"> <params> <param name="CMDS_TO_RUN" value="date"/> </params> </task> Check the output for /distribution/command in each guest. Expected results: Output should show an accurate clock in all cases. You can compare against the start time of the task shown in Beaker. Allow a few seconds leeway for harness overhead. Also bear in mind Beaker's UI shows your local time but the output will be in lab local time.
(In reply to Dan Callaghan from comment #16) ... and for the other half of this bug (re-enabling clock syncing in guests): Run a guest recipe with ks_meta="" and /distribution/reservesys, then log in and confirm that chrony (RHEL7) or ntpd (RHEL<=6) is running and clock has been synced.
(In reply to Dan Callaghan from comment #16) > Suggestions for verification: > Run a matrix of RHEL5, RHEL6, RHEL7 host x RHEL5, RHEL6, RHEL7 guest. > In the guest recipes set ks_meta="no_clock_sync", and run a command like > this: > <task name="/distribution/command"> > <params> > <param name="CMDS_TO_RUN" value="date"/> > </params> > </task> > Check the output for /distribution/command in each guest. The other thing to check is that /etc/adjtime has UTC and not LOCAL in all guests. You can check it in a similar way: <task name="/distribution/command"> <params> <param name="CMDS_TO_RUN" value="cat /etc/adjtime"/> </params> </task>
Beaker 19.2 has been released.