Created attachment 1573848 [details]
timer801_2

Hi Yanhui,
  Can I just check I understand the problem a bit more.
Looking at the screen shots each clock always seems to go forward - the clocks never go backwards.
But I think what you're saying is the difference between 'date' and 'hwclock' gets bigger (originally they were the same, but after migration there's about 1.5 minutes diffrence).

Is this the problem that you're reporting here?

Can you explain exactly how you did the migration steps?

(In reply to Dr. David Alan Gilbert from comment #2)
> Hi Yanhui,
>   Can I just check I understand the problem a bit more.
> Looking at the screen shots each clock always seems to go forward - the
> clocks never go backwards.
> But I think what you're saying is the difference between 'date' and
> 'hwclock' gets bigger (originally they were the same, but after migration
> there's about 1.5 minutes diffrence).
> 
> Is this the problem that you're reporting here?
> 

Yes, the 'hwclock' is slower than 'date'.

> Can you explain exactly how you did the migration steps?

Running "migrate -d tcp:ip:port"  in hmp monitor.

Hi,
  Can you try and reproduce this via libvirt please?  I'd like to try and figure out
if it's something a customer could hit.

Version-Release number of selected component (if applicable):
libvirt-5.0.0-9.module+el8.0.1+3240+dc659f51.x86_64
qemu-kvm-2.12.0-63.module+el8+2833+c7d6d092.x86_64
kernel-4.18.0-80.el8.x86_64

1. prepare 2 host, sync the two rhel8.0.1 hosts system time with ntp server
(hostA)# chronyd -q 'server clock.redhat.com iburst'
(hostB)# chronyd -q 'server clock.redhat.com iburst'

1. prepare a guest on hostA with the following clock xml
  <clock offset='variable' adjustment='31536000' basis='utc'>
    <timer name='rtc' track='wall'/>
  </clock>

2. make sure 

2. start the guest, check the qemu cmd line
# ps aux |grep qemu |grep rtc
...
-rtc base=2020-06-10T10:54:25,clock=host
...

3. check the whole guest qemu cmd line
# ps aux |grep qemu 
qemu     31513  106  0.6 1970672 53452 ?       Sl   06:54   0:05 /usr/libexec/qemu-kvm -name guest=avocado-vt-vm1,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-22-avocado-vt-vm1/master-key.aes -machine pc-i440fx-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off -cpu Haswell-noTSX,vme=on,ds=on,acpi=on,ss=on,ht=on,tm=on,pbe=on,dtes64=on,monitor=on,ds_cpl=on,vmx=on,smx=on,est=on,tm2=on,xtpr=on,pdcm=on,f16c=on,rdrand=on,arat=on,tsc_adjust=on,xsaveopt=on,pdpe1gb=on,abm=on -m 1024 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid 1afd42e6-2279-47c0-9829-c87535bea694 -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=33,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2020-06-10T10:54:25,clock=host -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device qemu-xhci,p2=15,p3=15,id=usb,bus=pci.0,addr=0x4 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive file=/mnt/nfs/lizhu/images/RHEL-8.0-x86_64-latest.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1,write-cache=on -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charua-c60aba6e-b6d8-448b-ab6e-8c7b5c29f359,fd=35,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charua-c60aba6e-b6d8-448b-ab6e-8c7b5c29f359,id=ua-c60aba6e-b6d8-448b-ab6e-8c7b5c29f359,name=org.qemu.guest_agent.0 -device usb-tablet,id=input0,bus=usb.0,port=1 -vnc 127.0.0.1:0 -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -object rng-random,id=objrng0,filename=/dev/urandom -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x8 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on


3. check the network status, and the time of the guest
[root@localhost ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever

[root@localhost ~]# timedatectl 
               Local time: Wed 2020-06-10 18:56:14 CST
           Universal time: Wed 2020-06-10 10:56:14 UTC
                 RTC time: Wed 2020-06-10 10:56:14
                Time zone: Asia/Shanghai (CST, +0800)
System clock synchronized: no
              NTP service: active
          RTC in local TZ: no

4. migrate the guest to hostB
# virsh migrate avocado-vt-vm1 --live --verbose qemu+ssh://10.73.72.90/system --migrateuri tcp://10.73.72.90
root.72.90's password: 
Migration: [100 %]

5. check the qemu-cmd line on hostB
# ps aux |grep qemu |grep rtc
....
 -rtc base=2020-06-10T10:56:58,clock=host
....

6. check the time of guest
[root@localhost ~]# timedatectl 
               Local time: Wed 2020-06-10 19:00:00 CST
           Universal time: Wed 2020-06-10 11:00:00 UTC
                 RTC time: Wed 2020-06-10 11:00:00
                Time zone: Asia/Shanghai (CST, +0800)
System clock synchronized: no
              NTP service: active
          RTC in local TZ: no



Can not reproduce it via libvirt.

(In reply to Lili Zhu from comment #6)
> Version-Release number of selected component (if applicable):
> libvirt-5.0.0-9.module+el8.0.1+3240+dc659f51.x86_64
> qemu-kvm-2.12.0-63.module+el8+2833+c7d6d092.x86_64

Hi lizhu,

Your qemu version is different with mine. Could you use the same qemu version?

> kernel-4.18.0-80.el8.x86_64
> 
> 1. prepare 2 host, sync the two rhel8.0.1 hosts system time with ntp server
> (hostA)# chronyd -q 'server clock.redhat.com iburst'
> (hostB)# chronyd -q 'server clock.redhat.com iburst'
> 
> 1. prepare a guest on hostA with the following clock xml
>   <clock offset='variable' adjustment='31536000' basis='utc'>
>     <timer name='rtc' track='wall'/>
>   </clock>
> 
> 2. make sure 
> 
> 2. start the guest, check the qemu cmd line
> # ps aux |grep qemu |grep rtc
> ...
> -rtc base=2020-06-10T10:54:25,clock=host
> ...
> 
> 3. check the whole guest qemu cmd line
> # ps aux |grep qemu 
> qemu     31513  106  0.6 1970672 53452 ?       Sl   06:54   0:05
> /usr/libexec/qemu-kvm -name guest=avocado-vt-vm1,debug-threads=on -S -object
> secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-22-avocado-
> vt-vm1/master-key.aes -machine
> pc-i440fx-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off -cpu
> Haswell-noTSX,vme=on,ds=on,acpi=on,ss=on,ht=on,tm=on,pbe=on,dtes64=on,
> monitor=on,ds_cpl=on,vmx=on,smx=on,est=on,tm2=on,xtpr=on,pdcm=on,f16c=on,
> rdrand=on,arat=on,tsc_adjust=on,xsaveopt=on,pdpe1gb=on,abm=on -m 1024
> -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid
> 1afd42e6-2279-47c0-9829-c87535bea694 -no-user-config -nodefaults -chardev
> socket,id=charmonitor,fd=33,server,nowait -mon
> chardev=charmonitor,id=monitor,mode=control -rtc
> base=2020-06-10T10:54:25,clock=host -no-shutdown -global
> PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device
> qemu-xhci,p2=15,p3=15,id=usb,bus=pci.0,addr=0x4 -device
> virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive
> file=/mnt/nfs/lizhu/images/RHEL-8.0-x86_64-latest.qcow2,format=qcow2,if=none,
> id=drive-virtio-disk0,cache=none -device
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,
> id=virtio-disk0,bootindex=1,write-cache=on -chardev pty,id=charserial0
> -device isa-serial,chardev=charserial0,id=serial0 -chardev
> socket,id=charua-c60aba6e-b6d8-448b-ab6e-8c7b5c29f359,fd=35,server,nowait
> -device
> virtserialport,bus=virtio-serial0.0,nr=1,chardev=charua-c60aba6e-b6d8-448b-
> ab6e-8c7b5c29f359,id=ua-c60aba6e-b6d8-448b-ab6e-8c7b5c29f359,name=org.qemu.
> guest_agent.0 -device usb-tablet,id=input0,bus=usb.0,port=1 -vnc 127.0.0.1:0
> -device
> qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,
> vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 -device
> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -object
> rng-random,id=objrng0,filename=/dev/urandom -device
> virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x8 -sandbox
> on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg
> timestamp=on
> 
> 
> 3. check the network status, and the time of the guest
> [root@localhost ~]# ip a
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group
> default qlen 1000
>     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
>     inet 127.0.0.1/8 scope host lo
>        valid_lft forever preferred_lft forever
>     inet6 ::1/128 scope host 
>        valid_lft forever preferred_lft forever
> 
> [root@localhost ~]# timedatectl 
>                Local time: Wed 2020-06-10 18:56:14 CST
>            Universal time: Wed 2020-06-10 10:56:14 UTC
>                  RTC time: Wed 2020-06-10 10:56:14
>                 Time zone: Asia/Shanghai (CST, +0800)
> System clock synchronized: no
>               NTP service: active
>           RTC in local TZ: no
> 
> 4. migrate the guest to hostB
> # virsh migrate avocado-vt-vm1 --live --verbose
> qemu+ssh://10.73.72.90/system --migrateuri tcp://10.73.72.90
> root.72.90's password: 
> Migration: [100 %]
> 
> 5. check the qemu-cmd line on hostB
> # ps aux |grep qemu |grep rtc
> ....
>  -rtc base=2020-06-10T10:56:58,clock=host
> ....
> 
> 6. check the time of guest
> [root@localhost ~]# timedatectl 
>                Local time: Wed 2020-06-10 19:00:00 CST
>            Universal time: Wed 2020-06-10 11:00:00 UTC
>                  RTC time: Wed 2020-06-10 11:00:00
>                 Time zone: Asia/Shanghai (CST, +0800)
> System clock synchronized: no
>               NTP service: active
>           RTC in local TZ: no
> 
> 
> 
> Can not reproduce it via libvirt.

Using Yama's hosts, I can't reproduce it with a simpler commandline:

/usr/libexec/qemu-kvm -M pc,accel=kvm -cpu host,vmx=off -m 8G -smp 4 -nographic -drive if=virtio,file=/mnt/rhel801-64-virtio.qcow2,cache=none -rtc base=2020-05-30,clock=host -net none

then in the guest I'm running:

while true; do date;hwclock; sleep 5; done &

and it's still consistent after migration.
I'm using:
  migrate_set_speed 100M
  migrate -d tcp:otherhost:4444

I'll try a commandline closer to Yama's now

I've reproduced something here using Yama's command line on Yama's hosts; but I don't understand it yet.

Directly after the migrate the two were still in sync; after a while though the RTC is ~31 seconds behind.
It doesn't feel like it was a slow slip, so it's not clear to me when it happened.
it also doesn't feel to me as if it's slipping more later.

I wasn't initially sure if the problem was that the software clock was fast or the hwclock was slow, it looks
like it is the hwclock is slow:

Some more testing on Yama's host:

16:27:15 real time shows as 08:01:25 in guest near boot
     so 25m50s offset
16:33:20 real time shows as 08:07:30 in guest later
16:34:15 real time shows as 08:08:23 in guest after migrate   (17 s migrate)
     so 25m52s offset ?
16:40:10 real time shows as
   08:14:13 date      25m57s
   08:13:54 hwclock   26m16s

so while the CPU clock slipped a few seconds, hwclock slipped much more.

I can repeat it without the +kvm_pv_unhalt

The offset I got seemed to be much less without most of the USB devices - but there was still ~2s offset.

(In reply to Yanhui Ma from comment #7)
> (In reply to Lili Zhu from comment #6)
> > Version-Release number of selected component (if applicable):
> > libvirt-5.0.0-9.module+el8.0.1+3240+dc659f51.x86_64
> > qemu-kvm-2.12.0-63.module+el8+2833+c7d6d092.x86_64
> 
> Hi lizhu,
> 
> Your qemu version is different with mine. Could you use the same qemu
> version?
> 
> > kernel-4.18.0-80.el8.x86_64
> > 
> > 1. prepare 2 host, sync the two rhel8.0.1 hosts system time with ntp server
> > (hostA)# chronyd -q 'server clock.redhat.com iburst'
> > (hostB)# chronyd -q 'server clock.redhat.com iburst'
> > 
> > 1. prepare a guest on hostA with the following clock xml
> >   <clock offset='variable' adjustment='31536000' basis='utc'>
> >     <timer name='rtc' track='wall'/>
> >   </clock>
> > 
> > 2. make sure 
> > 
> > 2. start the guest, check the qemu cmd line
> > # ps aux |grep qemu |grep rtc
> > ...
> > -rtc base=2020-06-10T10:54:25,clock=host
> > ...
> > 
> > 3. check the whole guest qemu cmd line
> > # ps aux |grep qemu 
> > qemu     31513  106  0.6 1970672 53452 ?       Sl   06:54   0:05
> > /usr/libexec/qemu-kvm -name guest=avocado-vt-vm1,debug-threads=on -S -object
> > secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-22-avocado-
> > vt-vm1/master-key.aes -machine
> > pc-i440fx-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off -cpu
> > Haswell-noTSX,vme=on,ds=on,acpi=on,ss=on,ht=on,tm=on,pbe=on,dtes64=on,
> > monitor=on,ds_cpl=on,vmx=on,smx=on,est=on,tm2=on,xtpr=on,pdcm=on,f16c=on,
> > rdrand=on,arat=on,tsc_adjust=on,xsaveopt=on,pdpe1gb=on,abm=on -m 1024
> > -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid
> > 1afd42e6-2279-47c0-9829-c87535bea694 -no-user-config -nodefaults -chardev
> > socket,id=charmonitor,fd=33,server,nowait -mon
> > chardev=charmonitor,id=monitor,mode=control -rtc
> > base=2020-06-10T10:54:25,clock=host -no-shutdown -global
> > PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device
> > qemu-xhci,p2=15,p3=15,id=usb,bus=pci.0,addr=0x4 -device
> > virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive
> > file=/mnt/nfs/lizhu/images/RHEL-8.0-x86_64-latest.qcow2,format=qcow2,if=none,
> > id=drive-virtio-disk0,cache=none -device
> > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,
> > id=virtio-disk0,bootindex=1,write-cache=on -chardev pty,id=charserial0
> > -device isa-serial,chardev=charserial0,id=serial0 -chardev
> > socket,id=charua-c60aba6e-b6d8-448b-ab6e-8c7b5c29f359,fd=35,server,nowait
> > -device
> > virtserialport,bus=virtio-serial0.0,nr=1,chardev=charua-c60aba6e-b6d8-448b-
> > ab6e-8c7b5c29f359,id=ua-c60aba6e-b6d8-448b-ab6e-8c7b5c29f359,name=org.qemu.
> > guest_agent.0 -device usb-tablet,id=input0,bus=usb.0,port=1 -vnc 127.0.0.1:0
> > -device
> > qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,
> > vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 -device
> > virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -object
> > rng-random,id=objrng0,filename=/dev/urandom -device
> > virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x8 -sandbox
> > on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg
> > timestamp=on
> > 
> > 
> > 3. check the network status, and the time of the guest
> > [root@localhost ~]# ip a
> > 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group
> > default qlen 1000
> >     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> >     inet 127.0.0.1/8 scope host lo
> >        valid_lft forever preferred_lft forever
> >     inet6 ::1/128 scope host 
> >        valid_lft forever preferred_lft forever
> > 
> > [root@localhost ~]# timedatectl 
> >                Local time: Wed 2020-06-10 18:56:14 CST
> >            Universal time: Wed 2020-06-10 10:56:14 UTC
> >                  RTC time: Wed 2020-06-10 10:56:14
> >                 Time zone: Asia/Shanghai (CST, +0800)
> > System clock synchronized: no
> >               NTP service: active
> >           RTC in local TZ: no
> > 
> > 4. migrate the guest to hostB
> > # virsh migrate avocado-vt-vm1 --live --verbose
> > qemu+ssh://10.73.72.90/system --migrateuri tcp://10.73.72.90
> > root.72.90's password: 
> > Migration: [100 %]
> > 
> > 5. check the qemu-cmd line on hostB
> > # ps aux |grep qemu |grep rtc
> > ....
> >  -rtc base=2020-06-10T10:56:58,clock=host
> > ....
> > 
> > 6. check the time of guest
> > [root@localhost ~]# timedatectl 
> >                Local time: Wed 2020-06-10 19:00:00 CST
> >            Universal time: Wed 2020-06-10 11:00:00 UTC
> >                  RTC time: Wed 2020-06-10 11:00:00
> >                 Time zone: Asia/Shanghai (CST, +0800)
> > System clock synchronized: no
> >               NTP service: active
> >           RTC in local TZ: no
> > 
> > 
> > 
> > Can not reproduce it via libvirt.

Tested with qemu-kvm-3.1.0-27.module+el8.0.1+3253+c5371cb3.x86_64
[root@localhost ~]# timedatectl 
               Local time: Thu 2020-06-11 10:27:09 CST
           Universal time: Thu 2020-06-11 02:27:09 UTC
                 RTC time: Thu 2020-06-11 02:27:09
                Time zone: Asia/Shanghai (CST, +0800)
System clock synchronized: no
              NTP service: active
          RTC in local TZ: no

Still can not reproduced

Hi dgilbert, 

Do you plan to fix it in 8.0.1? If yes, could you give devel_ack?

Thanks.

(In reply to Yanhui Ma from comment #13)
> Hi dgilbert, 
> 
> Do you plan to fix it in 8.0.1? If yes, could you give devel_ack?
> 
> Thanks.

I can't tell under I understand the cause.

Repeated on my own host, migrating to the same machine.
(virtlab413, 8.1 ish guest I installed myself).

It's got a few less cores on that host, so I'm using:

/usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1'  \
    -sandbox off  \
    -machine pc  \
    -nodefaults \
    -device VGA,bus=pci.0,addr=0x2  \
    -device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 \
    -device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 \
    -device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 \
    -device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 \
    -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=./rhel8-graphical.qcow2 \
    -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0,addr=0x3 \
    -m 4096  \
    -smp 10,cores=5,threads=1,sockets=2  \
    -cpu 'SandyBridge',+kvm_pv_unhalt \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :0  \
    -boot order=cdn,once=c,menu=off,strict=off \
    -rtc base=2020-05-30,clock=host \
    -enable-kvm -monitor stdio

Also seen on upstream head; (4.0.50 ish)
also happens with a rtc base in the past (i.e. 2019-05-30) - although only seen a second or two out not as big.

 My way of reproducing it is to :
   date; hwclock  on the source  - check they're the same
   migrate
   date;hwclock on the destination - still the same

  wait 2 minutes without using hte destination VM
   date;hwclock  again

Some other observations:
  rtc_update_timer isn't actually called regularly - even on the source
  qemu_clock_get_ns isn't called regularly either - which means that the notifier list for host clock reset is run
   if it's not called within 60 seconds - because it assumes the host clock jumped; but actually all that happened
is no one read it, so 'last' isn't updated.

I'm suspecting it needs one of these jumps to cause the problem, but am not sure why yet.

Hi,
  Can you retry your test with libvirt but, after migration please do;

     a) Check the time
     b) Wait for 3 minutes or more
     c) Check the time again.

Test with:
libvirt-5.0.0-11.module+el8.0.1+3459+e357ef2f.x86_64
qemu-kvm-3.1.0-27.module+el8.0.1+3253+c5371cb3.x86_64
kernel-4.18.0-80.el8.x86_64
Guest kernel: kernel-3.10.0-1058.el7.x86_64

1) Start a vm, down the nic and check the time:
# hwclock ;date
Wed 03 Jul 2019 12:19:05 PM CST  -0.909620 seconds
Wed Jul  3 12:19:04 CST 2019

2) Migrate vm

3) Check vm time:
# hwclock ;date
Wed 03 Jul 2019 12:19:50 PM CST  -0.685035 seconds
Wed Jul  3 12:19:49 CST 2019

4) After several mins, check vm time again:
# hwclock ;date
Wed 03 Jul 2019 12:26:31 PM CST  -1.002559 seconds
Wed Jul  3 12:26:30 CST 2019


Qemu command line:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin \
QEMU_AUDIO_DRV=spice \
/usr/libexec/qemu-kvm \
-name guest=nfs,debug-threads=on \
-S \
-object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-11-nfs/master-key.aes \
-machine pc-i440fx-rhel7.6.0,accel=kvm,usb=off,vmport=off,dump-guest-core=off \
-cpu IvyBridge \
-m 1000 \
-realtime mlock=off \
-smp 1,sockets=1,cores=1,threads=1 \
-uuid c1198dbc-90a3-4dab-b067-874ed991a502 \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=30,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=2019-07-03T04:15:37,clock=host \
-no-shutdown \
-global PIIX4_PM.disable_s3=1 \
-global PIIX4_PM.disable_s4=1 \
-boot strict=on \
-device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 \
-device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5 \
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1 \
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 \
-device usb-ccid,id=ccid0,bus=usb.0,port=2 \
-drive file=/mnt/RHEL-7.7-20190625.1-x86_64.qcow2,format=qcow2,if=none,id=drive-ide0-0-0,cache=none \
-device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1,write-cache=on \
-netdev tap,fd=32,id=hostnet0 \
-device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:7c:de:f1,bus=pci.0,addr=0x3 \
-chardev spicevmc,id=charsmartcard0,name=smartcard \
-device ccid-card-passthru,chardev=charsmartcard0,id=smartcard0,bus=ccid0.0 \
-chardev pty,id=charserial0 \
-device isa-serial,chardev=charserial0,id=serial0 \
-chardev spicevmc,id=charchannel0,name=vdagent \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 \
-chardev socket,id=charchannel1,fd=33,server,nowait \
-device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 \
-spice port=5900,addr=0.0.0.0,disable-ticketing,image-compression=off,seamless-migration=on \
-device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 \
-device intel-hda,id=sound0,bus=pci.0,addr=0x4 \
-device hda-micro,id=sound0-codec0,bus=sound0.0,cad=0 \
-chardev spicevmc,id=charredir0,name=usbredir \
-device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=1 \
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on

I met a different issue when test postcopy migration: vm system time is several seconds behind the expected time after migration.

Steps:
1) Start a vm and check the time:
# ssh 192.168.122.78 "hwclock&&date -u";date -u
Wed 03 Jul 2019 03:33:28 PM CST  -0.388983 seconds  ===> VM hardware time
Wed Jul  3 07:33:27 UTC 2019                        ===> VM system time
Wed Jul  3 05:33:28 UTC 2019                        ===> Host system time

2) Migrate vm via postcopy

3) Check time on dest host:
# ssh 192.168.122.78 "hwclock&&date -u";date -u
Wed 03 Jul 2019 03:35:19 PM CST  -0.134729 seconds  ===> VM hardware time
Wed Jul  3 07:35:05 UTC 2019                        ===> VM system time, 14 seconds behind
Wed Jul  3 05:35:19 UTC 2019                        ===> Host system time

4) After a several mins, check the time again, vm system time is still behind:
# ssh 192.168.122.78 "hwclock&&date -u";date -u
Wed 03 Jul 2019 03:42:13 PM CST  -1.002220 seconds
Wed Jul  3 07:41:59 UTC 2019
Wed Jul  3 05:42:13 UTC 2019

I can trigger this with rhel7 guests - so it's not related to the new guests
I can also trigger it with a 2.6.2 qemu; so it's not related to a new qemu.

So it looks like this has been around for ages.

*** Bug 1719081 has been marked as a duplicate of this bug. ***

OK, this is subtle, but the way libvirt drives it means that this isn't a bug directly; however I'm going to try
and improve qemu's docs and may change a couple of other things.


1) QEMU calculates an offset from the host clock at startup using the options to -rtc
2) It only really uses this during initialisation - especially on anything other than x86
3) There's a mechanism to detect host clock jumps; it's a bit broken, the easiest way to trigger it is to read the clock
  with hwclock, wait 60 seconds and then read it again.  When that happens the rtc gets recalculated from the host
  clock and uses the offset again.
4) When we migrate, if you set base= the same date on the source and destination then the offset on the destination is
inconsistent with the migrated time;  this is the problem we see here.  The amount hwclock is seen to jump

5) Libvirt calculates the base using an 'adjustment' from the current time and recalculates that on the destination;
that means libvirt sets -rtc base= on the destination different from the source.

6) The difference libvirt uses is the difference in the time the source and destination were started; and that's exactly
the error we get in this bz - so this cancels out and the problem doesn't happen with libvirt.