Created attachment 964096 [details]
serial log for stop/continue

Find similar problems when test it with stop/continue operation.

Just change the step 2 to stop guest for 5 minutes. And then continue it.

The attachment is the Call trace get from serial log.

Hi, thanks for the report.  Can a clarify a few things:

1) What method did you use to reboot the guest?  Was it shutdown then restarted using virsh, or just rebooted with qemu still running?

If the guest was shutdown, can you show the output from virsh dumpxml between the first shutdown and the restart - in this case I'd expect the offset='utc' to be automatically changed to offset='variable'

2) What's the qemu command line which your libvirt XML produces?

3) To clarify what the incorrect behaviour is here, I believe you're expecting the RTC to only measure time spent within the guest because of the track='guest'.  Is that correct?



NOTE: The 'pit' and 'hpet' clauses should be omitted on Power - they control PC platform specific devices and will have no effect for a Power guest.

(In reply to David Gibson from comment #4)

1) I rebooted the guest inside the guest with cmd "reboot".

2) Following is the qemu command line that I used: For the BE guest named 'rhel7.1', the libvirt xml segment is '<timer name='rtc' tickpolicy='catchup' track='guest'/>'; for the LE guest named 'rhel7.1-le', the libvirt xml segment is '<timer name='rtc' tickpolicy='delay' track='guest'/>'

[root@ibm-p8-kvm-01-qe ngu]# ps aux|grep qemu
qemu      84738  0.2  0.9 20405312 1275776 ?    Sl   05:43   2:08 /usr/libexec/qemu-kvm -name rhel7.1 -S -machine pseries,accel=kvm,usb=off -m 16384 -realtime mlock=off -smp 32,sockets=1,cores=4,threads=8 -uuid 95346a10-1828-403a-a610-ac5a52a29479 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/rhel7.1.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,clock=vm,driftfix=slew -no-shutdown -boot strict=on -device usb-ehci,id=usb,bus=pci.0,addr=0x2 -device pci-ohci,id=usb1,bus=pci.0,addr=0x1 -device spapr-vscsi,id=scsi0,reg=0x1000 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x3 -drive file=/home/ngu/rhel7.1.qcow2,if=none,id=drive-scsi0-0-0-0,format=qcow2,cache=none -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 -netdev tap,fd=27,id=hostnet0 -device spapr-vlan,netdev=hostnet0,id=net0,mac=52:54:00:c4:e7:19,reg=0x2000 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/rhel7.1.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -chardev socket,id=charconsole0,path=/var/lib/libvirt/qemu/rhel7.1-serial.sock,server,nowait -device virtconsole,bus=virtio-serial0.0,nr=1,chardev=charconsole0,id=console0 -device usb-kbd,id=input0 -device usb-mouse,id=input1 -vnc 0:7 -device VGA,id=video0,bus=pci.0,addr=0x4 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -object rng-random,id=rng0,filename=/dev/random -device virtio-rng-pci,rng=rng0,max-bytes=1234,period=2000,bus=pci.0,addr=0x6 -msg timestamp=on

qemu     144527  7.8  0.6 20339008 883200 ?     Sl   20:40   0:32 /usr/libexec/qemu-kvm -name rhel7.1-le -S -machine pseries,accel=kvm,usb=off -m 16384 -realtime mlock=off -smp 32,sockets=1,cores=4,threads=8 -uuid 95346a10-1828-403a-a610-ac5a52a29472 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/rhel7.1-le.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,clock=vm -no-shutdown -boot strict=on -device usb-ehci,id=usb,bus=pci.0,addr=0x2 -device pci-ohci,id=usb1,bus=pci.0,addr=0x1 -device spapr-vscsi,id=scsi0,reg=0x1000 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x3 -drive file=/home/ngu/rhel7.1-le.qcow2,if=none,id=drive-scsi0-0-0-0,format=qcow2,cache=none -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 -netdev tap,fd=25,id=hostnet0 -device spapr-vlan,netdev=hostnet0,id=net0,mac=52:54:00:c4:e7:12,reg=0x2000 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/rhel7.1-le.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -chardev socket,id=charconsole0,path=/var/lib/libvirt/qemu/rhel7.1-le-serial.sock,server,nowait -device virtconsole,bus=virtio-serial0.0,nr=1,chardev=charconsole0,id=console0 -device usb-kbd,id=input0 -device usb-mouse,id=input1 -vnc 0:6 -device VGA,id=video0,bus=pci.0,addr=0x4 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -object rng-random,id=rng0,filename=/dev/random -device virtio-rng-pci,rng=rng0,max-bytes=1234,period=2000,bus=pci.0,addr=0x6 -msg timestamp=on

3) That's right.

Ah, yes, it looks like the pseries RTAS real time clock implementation doesn't honour the -rtc clock=vm option to qemu.

It will take me a little while to address this one.

Created attachment 966607 [details]
qemu-kvm-rhev-2.1.2-16.el7.test.ppc64.rpm

Created attachment 966622 [details]
qemu-kvm-common-rhev-2.1.2-16.el7.test.ppc64.rpm

Created attachment 966623 [details]
qemu-img-rhev-2.1.2-16.el7.test.ppc64.rpm

Created attachment 966624 [details]
qemu-kvm-rhev-debuginfo-2.1.2-16.el7.test.ppc64.rpm

I'm having problems brew building again, so I've uploaded some manually built packages which have a draft fix for this bug.

I believe this should stop the guest rtc changing when the host time is changed if clock=vm is specified.  I'm less sure about the suspend case.

Can you please try these out.  Note that these patches are a work in progress and will need some upstream review before they're ready to merge.

------- Comment From fnovak.com 2014-12-10 13:30 EDT-------
reverse mirror of RHBZ Bug 1170132 - Guest time could change with host time even specify the guest clock as "-rtc base=utc,clock=vm,..."

RFC patch series posted upstream.

See http://lists.nongnu.org/archive/html/qemu-devel/2014-12/msg01452.html

I have now successfully brewed the draft patches at:

http://brewweb.devel.redhat.com/brew/taskinfo?taskID=8363604

(In reply to David Gibson from comment #11/14)

Have upgrade the patches to do test, no the bug problem any more, i.e. guest time would not change with the host time when set guest clock as "-rtc clock=vm,...".

Thanks for the feedback.

Have you tested stopping / continuing the guest as well as changing the host time?

I am currently working towards getting this fix upstream.

Created attachment 967603 [details]
hwclock lags or rolls back after stop&cont 2014-12-12 19:53:54.png

(In reply to David Gibson from comment #16)
> Thanks for the feedback.
> 
> Have you tested stopping / continuing the guest as well as changing the host
> time?
> 
> I am currently working towards getting this fix upstream.

Have done some stop/cont test on the test package https://brewweb.devel.redhat.com/taskinfo?taskID=8368750, met guest hwclock lagging or rolling back issue with following step if booted the guest with "-rtc base=2006-06-06,clock=vm":

1.Check the guest system time and hwclock
2.Stop the guest with qemu cmd "stop"
3.Change the host system time 1 hour ahead
4.Wait for 3-5 mins, then resume the guest with qemu cmd "cont"
5.Check the guest system time and hwclock again, then it's found the hwclock ticked not as right freqency.

This happens on both BE and LE guest.

Sorry, I'm not sure what you mean by "5.Check the guest system time and hwclock again, then it's found the hwclock ticked not as right freqency."

(In reply to David Gibson from comment #18)

That's I found the hwclock lagged or rolled back compared with that in step 1.

That's the expected behaviour then, isn't it?  With clock=vm, the guest RTC should not advance while the guest is stopped.

(In reply to David Gibson from comment #20)
> That's the expected behaviour then, isn't it?  With clock=vm, the guest RTC
> should not advance while the guest is stopped.

David,

You are right, it's my fault. Thanks.

------- Comment From seg.com 2015-01-12 21:13 EDT-------
So, is the conclusion that this is not a bug? If that's the case, we should close. If not, can someone please summarize where we are with this bug? It's become hard to follow.

No, the conclusion is that my draft patches appear to be correctly fix the problem.  As noted in bug 1171700 though, they're not upstream yet.

Also found the problem on ibm power system:

Host kernel: 3.10.53-2020.1.pkvm2_1_1.49.ppc64/3.10.42-2018.1.pkvm2_1_1.46.ppc64
Guest kernel: 3.10.0-223.el7.ppc64/3.10.0-223.ael7b.ppc64le
Qemu-kvm version:
qemu-img-2.0.0-2.1.pkvm2_1_1.20.40.ppc64
qemu-kvm-2.0.0-2.1.pkvm2_1_1.20.40.ppc64
qemu-common-2.0.0-2.1.pkvm2_1_1.20.40.ppc64
qemu-system-ppc-2.0.0-2.1.pkvm2_1_1.20.40.ppc64
qemu-kvm-tools-2.0.0-2.1.pkvm2_1_1.20.40.ppc64
qemu-system-x86-2.0.0-2.1.pkvm2_1_1.20.40.ppc64
qemu-2.0.0-2.1.pkvm2_1_1.20.40.ppc64

Gu Nini,

Thanks for the confirmation.  That was expected, since it is an upstream bug.  I've been busy and haven't had a chance to resend my upstream patches; I'm hoping I'll be able to do that today.

Draft patches posted upstream, see https://www.mail-archive.com/qemu-devel@nongnu.org/msg271868.html

Still under discussion.

Fix included in qemu-kvm-rhev-2.2.0-8.el7

Verify the bug on both ppc64be and ppc64le host with following sw versions:

PPC64BE HOST:
host kernel: 3.10.0-300.el7.ppc64
Guest kernel: 3.10.0-295.el7.ppc64/3.10.0-295.el7.ppc64le
Qemu-kvm-rhev: qemu-kvm-rhev-2.3.0-13.el7.ppc64

PPC64LE HOST:
host kernel: 3.10.0-300.el7.ppc64le
Guest kernel: 3.10.0-295.el7.ppc64/3.10.0-295.el7.ppc64le
Qemu-kvm-rhev: qemu-kvm-rhev-2.3.0-13.el7.ppc64le


Now when set the guest clock as "-rtc base=utc,clock=vm" or "-rtc base=2006-06-06,clock=vm", the guest time would not change with the host time with the steps in the bug.

However, I should re-emphasize the problem in comments 17-21 with following steps:

1) Start a guest with guest clock "-rtc base=utc,clock=vm" or "-rtc base=2006-06-06,clock=vm"
2) After the guest boots up, check its system time and hwclock
3) 'stop' the guest, then change the host system time with cmd 'date -s XX:XX:XX'
4) Wait for 5 mins, then 'cont' the guest, and check its system time and hwclock

Then it's found the guest hwclock would tick with the same frenqency as the host system time, i.e., if I changed the host system time one hour behind, it would tick quicker to chase up the time server(maybe), then the guest hwclock would ticks quicker too, I can check this from the time difference between guest system time and hwclock; if I changed the host system time one hour ahead, it would tick slower to wait for the time server, then the guest hwclock would ticks slower too, so I had found the guest hwclock lags or rolls back as showed in comment 17.

David,

Would you help to check if this is a problem and should be report a new bug?

Thanks!

Nini

Hi Nini,

Thanks for the description.  I think I didn't properly understand the problem the first time you mentioned it.  Basically, with clock=vm the guest is unaffected by sudden jumps in the host time, but *is* affected by gradual slew adjustments in the host.  This makes sense, because the guest hwclock is based on the host's CLOCK_MONOTONIC, which is unaffected by discontinuous changes in the host time, but is affected by slew adjustments.

Exactly what the semantics of clock=vm should be in this case is unclear, but since it is described as having the guest clock disconnected from the host, it makes some sense that it not be affected by host slew adjustments.

I think it would be possible to fix this using the kernel's CLOCK_MONOTONIC_RAW time source, however doing this would require substantial rework of qemu's core timing code: as far as I can tell the behaviour will be the same on x86.

I don't think this is a problem in practice; it only shows up in a rather contrived edge case.  If you like, you can file a new BZ to track it, but I don't expect it to be changed for RHEL 7.2, and may end up closed as WONTFIX.

(In reply to David Gibson from comment #32)

David,

Thanks for the detailed explanation. I reported bz 1248860 to track the issue in comment 17-21/31-32, I agree with you that can close the bug as WONTFIX, but just use it as a problem tracker.

Setting to VERIFIED according to comment 30.

------- Comment From seg.com 2015-08-28 04:34 EDT-------
Based on where we are in the product cycle and because of the potential for affecting existing customers, we are going to defer fixing this to the next product version.

Hi David, 

I found some interesting result in my test according to comment #20

Test version:
Host:3.10.0-313.el7.ppc64le
Qemu:qemu-kvm-rhev-2.3.0-22.el7
Guest:3.10.0-229.14.1.ael7b.ppc64le

step:
1. Boot up guest with "-rct base=utc,clock=vm"
2. Check the guest time 
Guest: #date
       Tue Sep 15 22:47:38 EDT 2015

3. Do this step Immediately after step 2. Stop guest in HMP
Host: (qemu)stop

4. After 2-3 minutes, continue guest , and check time in guest
Host: (qemu)cont
Guest: #date
       Tue Sep 15 22:50:40 EDT 2015


Seems the time is still advancing while guest status is paused. Is this normal or a bug ?

Zhengtong,

With clock=vm the clock should not advance while the guest is paused, so this is another bug.  Please file a new BZ for this - target it for RHEL7.3, this isn't important enough to fix for RHEL 7.2.

(In reply to David Gibson from comment #37)
I have filed a new bug about this issue.

https://bugzilla.redhat.com/show_bug.cgi?id=1264258

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2546.html