Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 877930

Summary: Windows server 2012 guest system does not auto adjust time after resume from paused status
Product: Red Hat Enterprise Linux 6 Reporter: Qunfang Zhang <qzhang>
Component: qemu-kvmAssignee: Gleb Natapov <gleb>
Status: CLOSED NOTABUG QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.4CC: acathrow, areis, bsarathy, dyasny, fyang, gleb, juzhang, knoel, michen, mkenneth, mtosatti, virt-maint, xwei
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-06-05 12:09:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
ftrace log before cont
none
ftrace just before cont guest
none
ftrace collected with trace-cmd none

Description Qunfang Zhang 2012-11-19 09:28:11 UTC 
Description of problem:
Sync host time with a clock server, like clock.redhat.com.
Boot a windows server 2012 guest and sync the time with the same clock server. Then stop it using the 'stop' qemu monitor command and record the time. Resume the guest after 5 minutes with 'cont' command.
Watching the guest time, and found the guest doesn't auto adjust the time. It will be 5 minutes backwards.
Test win2k8-r2 guest with the same command line, this issue doesn't exist. Win2k8-r2 guest will auto adjust time after resume from paused status.

Version-Release number of selected component (if applicable):
kernel-2.6.32-342.el6.x86_64
qemu-kvm-0.12.1.2-2.334.el6.x86_64

How reproducible:
Always

Steps to Reproduce:
0. Sync host time with clock.redhat.com
#ntpdate clock.redhat.com

1. Boot a windows 2012 guest:
/usr/libexec/qemu-kvm -boot menu=on -m 2G -smp 2,cores=2,sockets=1,threads=1 -M rhel6.4.0 -cpu SandyBridge -drive file=/mnt/win2012-64-virtio.qcow2,format=qcow2,if=none,id=drive-ide0,cache=none,werror=stop,rerror=stop  -device virtio-blk-pci,bus=pci.0,addr=0x3,drive=drive-ide0,id=test0  -netdev tap,id=hostnet1,vhost=on,script=/etc/qemu-ifup,downscript=no -device virtio-net-pci,netdev=hostnet1,mac=00:12:1a:21:62:02,bus=pci.0,addr=0x6,id=virtio-net-pci1 -uuid ac64c74a-a8d5-4c24-9839-fcc491439493 -rtc base=localtime,clock=host,driftfix=slew -no-kvm-pit-reinjection  -name rhel6.4 -monitor stdio -device virtio-balloon-pci,bus=pci.0,addr=0x8,id=balloon0 -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x9  -chardev socket,id=charchannel0,path=/tmp/qzhang-test,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -boot menu=on -boot c -drive if=none,werror=stop,rerror=stop,media=cdrom,id=drive-cdrom -device ide-drive,drive=drive-cdrom,id=cdrom -vnc :10  -usb -device usb-tablet 


2. Sync guest time with clock.redhat.com

3. (qemu) stop

4. After 5 minutes, 
(qemu) cont

5. Watching the guest time.

  
Actual results:
Guest doesn't auto adjust the time after resume from paused status.

Expected results:
Guest should auto adjust the time.

Additional info:
Win2k8-r2 guest has no problem with the same command line.
Comment 2 Marcelo Tosatti 2012-11-22 23:34:20 UTC 
I suppose Win2k8-r2 is synchronizing the time via w32time. Can you check by

1. Enabling w32time logging with 

w32tm /debug /enable /file:C:logstimelog.txt /size:100000 /entries:0-300

2. stop
wait 5 minutes
3. cont

Then checking logstimelog.txt (attach it please) ?

CCing Gleb.

For a guarantee of proper time sync (without possibility of an application reading stale time), interaction with the guest is required (such as S3-suspend-before-stop and S3-resume-before-cont).
Comment 3 Qunfang Zhang 2012-11-23 07:01:54 UTC 
(In reply to comment #2)
> I suppose Win2k8-r2 is synchronizing the time via w32time. Can you check by
> 
> 1. Enabling w32time logging with 
> 
> w32tm /debug /enable /file:C:logstimelog.txt /size:100000 /entries:0-300
> 
> 2. stop
> wait 5 minutes
> 3. cont
> 
> Then checking logstimelog.txt (attach it please) ?
> 
> CCing Gleb.
> 
> For a guarantee of proper time sync (without possibility of an application
> reading stale time), interaction with the guest is required (such as
> S3-suspend-before-stop and S3-resume-before-cont).

Hi, Marcelo
I test with your step 1-3, but after that, the log file logstimelog.txt is empty, do you know what's the reason? I will check it as well.  Thanks.
Comment 4 Gleb Natapov 2012-11-25 15:04:35 UTC 
(In reply to comment #2)
> For a guarantee of proper time sync (without possibility of an application
> reading stale time), interaction with the guest is required (such as
> S3-suspend-before-stop and S3-resume-before-cont).

With the given command line qemu supposed to reinject missed RTC interrupts untill time in a guest will catch up. Can you run ftrace just before cont?
Comment 5 Qunfang Zhang 2012-11-26 05:18:46 UTC 
Created attachment 651742 [details]
ftrace log before cont
Comment 6 Qunfang Zhang 2012-11-26 05:35:37 UTC 
Created attachment 651759 [details]
ftrace just before cont guest
Comment 7 Qunfang Zhang 2012-11-26 05:39:37 UTC 
(In reply to comment #4)
> (In reply to comment #2)
> > For a guarantee of proper time sync (without possibility of an application
> > reading stale time), interaction with the guest is required (such as
> > S3-suspend-before-stop and S3-resume-before-cont).
> 
> With the given command line qemu supposed to reinject missed RTC interrupts
> untill time in a guest will catch up. Can you run ftrace just before cont?

Hi, Gleb
Please check the second log I upload in comment 6. I collect the log just before cont guest.  The fist log is collected during the guest always stops.
Comment 8 Gleb Natapov 2012-11-26 08:28:35 UTC 
(In reply to comment #7)
> (In reply to comment #4)
> > (In reply to comment #2)
> > > For a guarantee of proper time sync (without possibility of an application
> > > reading stale time), interaction with the guest is required (such as
> > > S3-suspend-before-stop and S3-resume-before-cont).
> > 
> > With the given command line qemu supposed to reinject missed RTC interrupts
> > untill time in a guest will catch up. Can you run ftrace just before cont?
> 
> Hi, Gleb
> Please check the second log I upload in comment 6. I collect the log just
> before cont guest.  The fist log is collected during the guest always stops.

You should have traced KVM events "trace-cmd record -e kvm".
Comment 9 Qunfang Zhang 2012-11-26 10:11:11 UTC 
Created attachment 651858 [details]
ftrace collected with trace-cmd
Comment 10 Gleb Natapov 2012-11-26 10:23:39 UTC 
Does this happen with -smp 1?
Comment 11 Qunfang Zhang 2012-11-26 10:40:46 UTC 
(In reply to comment #10)
> Does this happen with -smp 1?

Yup, this happens with -smp 1 as well.
Comment 12 Marcelo Tosatti 2012-11-26 20:51:11 UTC 
(In reply to comment #4)
> (In reply to comment #2)
> > For a guarantee of proper time sync (without possibility of an application
> > reading stale time), interaction with the guest is required (such as
> > S3-suspend-before-stop and S3-resume-before-cont).
> 
> With the given command line qemu supposed to reinject missed RTC interrupts
> untill time in a guest will catch up. Can you run ftrace just before cont?

There's a limit on the number of reinjections per interrupt period. So reinjection 
is a "best attempt" kind of solution. There is no guarantee app will not see stale time.
Comment 13 Gleb Natapov 2012-11-27 07:37:30 UTC 
(In reply to comment #12)
> (In reply to comment #4)
> > (In reply to comment #2)
> > > For a guarantee of proper time sync (without possibility of an application
> > > reading stale time), interaction with the guest is required (such as
> > > S3-suspend-before-stop and S3-resume-before-cont).
> > 
> > With the given command line qemu supposed to reinject missed RTC interrupts
> > untill time in a guest will catch up. Can you run ftrace just before cont?
> 
> There's a limit on the number of reinjections per interrupt period. So
> reinjection 
> is a "best attempt" kind of solution. There is no guarantee app will not see
> stale time.

The limit exists, but if not all interrupts are re-injected injection rate is increased, so eventually all lost ticks should be reinjected. And of course while
reinjection happens guest will see incorrect time. Qunfang how long do you wait
after continue and before checking time inside the guest? Is time in the guest still incorrect 1min after cont? 5min?
Comment 14 Qunfang Zhang 2012-11-27 08:15:54 UTC 
(In reply to comment #13)
> (In reply to comment #12)
> > (In reply to comment #4)
> > > (In reply to comment #2)
> > > > For a guarantee of proper time sync (without possibility of an application
> > > > reading stale time), interaction with the guest is required (such as
> > > > S3-suspend-before-stop and S3-resume-before-cont).
> > > 
> > > With the given command line qemu supposed to reinject missed RTC interrupts
> > > untill time in a guest will catch up. Can you run ftrace just before cont?
> > 
> > There's a limit on the number of reinjections per interrupt period. So
> > reinjection 
> > is a "best attempt" kind of solution. There is no guarantee app will not see
> > stale time.
> 
> The limit exists, but if not all interrupts are re-injected injection rate
> is increased, so eventually all lost ticks should be reinjected. And of
> course while
> reinjection happens guest will see incorrect time. Qunfang how long do you
> wait
> after continue and before checking time inside the guest? Is time in the
> guest still incorrect 1min after cont? 5min?

Gleb
I checked again to cont guest after 1min, 3min and 5min, all can reproduce the issue.
Comment 15 Marcelo Tosatti 2012-11-27 21:25:53 UTC 
Moving to 6.5 as its not a customer report (also, assigned to Gleb).
Comment 19 Gleb Natapov 2013-05-02 10:46:13 UTC 
So starting from Windows2012 Windows no longer relies on number of injected RTC interrupts for time keeping. It reads PM timer on each time interrupt instead (hopefully it uses faster method if Hyper-V is available). This is actually more virtualization friendly. We can drop stupid RTC interrupt reinjections for Windows2012.

The time drift is seen because "stop" monitor command also stops PM timer. We are not going to fix that since this is unsupported scenario. What I noticed though is that Windows2012 is not able to fix time drift if PM timer overflows while vcpu is not running. Currently QEMU emulates 24bit PM timer and it overflows every 4.6 seconds, we should fix it to be 32bit to make the problem impossible to hit on practice.