Amos,

Can you remove the -no-kvm-pit-reinjection flag?

Err, ignore comment #2.

Can you confirm if the problem is present with -smp 1?

(In reply to comment #3)
> Err, ignore comment #2.
> 
> Can you confirm if the problem is present with -smp 1?    

This problem also exists with -smp 1.

After ping migration:
[root@localhost root]# ntpdate -q clock.redhat.com
server 66.187.233.4, stratum 1, offset 4.507959, delay 0.30522
 2 Mar 21:42:34 ntpdate[1314]: step time server 66.187.233.4 offset 4.507959 sec

After pong migration:
[root@localhost root]# ntpdate -q clock.redhat.com
server 66.187.233.4, stratum 1, offset 233.897184, delay 0.30522
 2 Mar 21:51:05 ntpdate[1321]: step time server 66.187.233.4 offset 233.897184 sec

Just an update: I'm blocked on getting ISO images over slow link... but 90% there now.

Amos, it might be an issue with the RTC emulation too, what's the diff when you reboot a rhel3 VM?

it would also be most helpful to see /proc/cpuinfo from the host, can we get this information please?

Also, can you check the ntpdate without doing any migration?  Just use a reliable clock, measure 30 seconds, check guest date, wait 30 seconds, check again - and see if guest advances by 30 seconds or jumps ahead and then back.

(In reply to comment #6)
> Amos, it might be an issue with the RTC emulation too, what's the diff when you
> reboot a rhel3 VM?    

I have tested 5 times of reboot rhel3 VM, the drift time are all less than 1 sec.

Result:
offset -0.876304 sec
offset -0.526459 sec
offset -0.797713 sec
offset -0.525171 sec
offset -0.869227 sec

(In reply to comment #7)
> it would also be most helpful to see /proc/cpuinfo from the host, can we get
> this information please?

I had uploaded cpuinfo and meminfo of host here:
https://bugzilla.redhat.com/attachment.cgi?id=396485

> Also, can you check the ntpdate without doing any migration?  Just use a
> reliable clock, measure 30 seconds, check guest date, wait 30 seconds, check
> again - and see if guest advances by 30 seconds or jumps ahead and then back.    

The change is very little without migration.

steps:
1. boot a rhel3 VM
2. get guest IP address
3. execute below script
-----------------------------------------
#!/bin/bash

time=$1
guest_ip="10.66.83.68"

sync_guest_ntpdate() {
    ssh $guest_ip ntpdate clock.redhat.com
}
get_guest_ntpdate() {
    ssh $guest_ip ntpdate -q clock.redhat.com
}

ssh-copy-id $guest_ip ~/.ssh/id_rsa.pub > /dev/null

sync_guest_ntpdate
sync_guest_ntpdate
echo "---------------"
get_guest_ntpdate
sleep $time
echo "---------------"
get_guest_ntpdate
sleep $time
echo "---------------"
get_guest_ntpdate
-----------------------------------------


Result:
1) Set sleep time to 30 sec
# ./timedrift.sh 30
10 Mar 23:59:17 ntpdate[2089]: adjust time server 66.187.233.4 offset -0.040228 sec
10 Mar 23:59:19 ntpdate[2098]: adjust time server 66.187.233.4 offset -0.036673 sec
---------------
server 66.187.233.4, stratum 1, offset -0.034486, delay 0.29512
10 Mar 23:59:20 ntpdate[2107]: adjust time server 66.187.233.4 offset -0.034486 sec
---------------
server 66.187.233.4, stratum 1, offset -0.024299, delay 0.30476
10 Mar 23:59:52 ntpdate[2116]: adjust time server 66.187.233.4 offset -0.024299 sec
---------------
server 66.187.233.4, stratum 1, offset -0.005983, delay 0.30484
11 Mar 00:00:23 ntpdate[2130]: adjust time server 66.187.233.4 offset -0.005983 sec


2) Set sleep time to 300 sec
# ./timedrift.sh 300
11 Mar 00:03:32 ntpdate[2159]: adjust time server 66.187.233.4 offset 0.008697 sec
11 Mar 00:03:33 ntpdate[2168]: adjust time server 66.187.233.4 offset 0.003830 sec
---------------
server 66.187.233.4, stratum 1, offset 0.007391, delay 0.29539
11 Mar 00:03:35 ntpdate[2177]: adjust time server 66.187.233.4 offset 0.007391 sec
---------------
server 66.187.233.4, stratum 1, offset -0.014029, delay 0.30519
11 Mar 00:08:36 ntpdate[2188]: adjust time server 66.187.233.4 offset -0.014029 sec
---------------
server 66.187.233.4, stratum 1, offset -0.027170, delay 0.30522
11 Mar 00:13:37 ntpdate[2202]: adjust time server 66.187.233.4 offset -0.027170 sec

I can't reproduce on Intel hardware with kvm-83-164.el5 

Can you try upgrading your kvm and see if it still happens?

I am not using -rtc-td-hack however, let me try it and see if it matters.

-rtc-td-hack makes no difference.  I don't see any drift after migration.

Does this still happen?  All attempts I have made at it have been unable to reproduce the bug.

(In reply to comment #12)
> Does this still happen?  All attempts I have made at it have been unable to
> reproduce the bug.    

I'm testing with kvm-83-170.el5 for 50 times  (Intel & AMD)
The result will be reported later.

(In reply to comment #12)
> Does this still happen?  All attempts I have made at it have been unable to
> reproduce the bug.    

Kernel: 2.6.18-194.el5
# rpm -qa |grep kvm
kvm-83-170.el5
etherboot-zroms-kvm-5.4.4-13.el5
kvm-qemu-img-83-170.el5
kvm-debuginfo-83-170.el5
kmod-kvm-83-170.el5
kvm-tools-83-170.el5


Result:

---- Host mem:4G smp:4 "AMD Phenom(tm) 9600B Quad-Core Processor"
Guest mem:4G smp:4 RHEL-39-64: always reproduce
Guest mem:4G smp:4 RHEL-39-32: always reproduce

---- Host mem:4G smp:4 "Intel(R) Core(TM)2 Quad CPU    Q9400  @ 2.66GHz"
Guest mem:4G smp:4 RHEL-39-64: always could not reproduce
Guest mem:4G smp:4 RHEL-39-32: always reproduce


I also test on other machines, it's easy to reproduce when guest's memory size is same as host's.

cpuinfo and meminfo attached.
# tree
.
|-- amd
|   |-- cpuinfo
|   `-- meminfo
`-- intel
    |-- cpuinfo
    `-- meminfo

Created attachment 407521 [details]
cpuinfo and meminfo of host (AMD & Intel)

this is the cpuinfo & meminfo of the hosts (amd & intel)

(In reply to comment #14)
> I also test on other machines, it's easy to reproduce when guest's memory size
> is same as host's.

That's almost certainly why the bug happens; likely that we are hitting swap on the host and one of the qemu processes gets hosed.

It's also probable that this scenario is restricted to self migration.  Note that self migration requires

1) reading all memory from old VM
2) passing data over network socket
3) writing all memory to new VM

this strides over the entire host memory and requires easily > 2X the host memory plus resources for network and processes.  You will swap, and performance will be bad.

I suggest this is not a migration bug at all.  Solving this problem is exactly equivalent to solving the problem where a user issues SIGSTOP to the qemu process for ~90 seconds.

Does anyonne know if RHEL 3.9 has tick compensation or not?  (It is 2.4, so I guess it does not).

Perhaps dropping -kvm-no-pit-reinjection is all that is needed to solve the bug.

Can you try that and see if the bug goes away?

(In reply to comment #16)
> (In reply to comment #14)
> > I also test on other machines, it's easy to reproduce when guest's memory size
> > is same as host's.
> 
> That's almost certainly why the bug happens; likely that we are hitting swap on
> the host and one of the qemu processes gets hosed.
> 
> It's also probable that this scenario is restricted to self migration.  Note
> that self migration requires
> 
> 1) reading all memory from old VM
> 2) passing data over network socket
> 3) writing all memory to new VM
> 
> this strides over the entire host memory and requires easily > 2X the host
> memory plus resources for network and processes.  You will swap, and
> performance will be bad.

yes, the performance is very bad.

> I suggest this is not a migration bug at all.  Solving this problem is exactly
> equivalent to solving the problem where a user issues SIGSTOP to the qemu
> process for ~90 seconds.
> 
> Does anyonne know if RHEL 3.9 has tick compensation or not?  (It is 2.4, so I
> guess it does not).
> 
> Perhaps dropping -kvm-no-pit-reinjection is all that is needed to solve the
> bug.
> 
> Can you try that and see if the bug goes away?    

Do offline migration, RESULT:
with '-kvm-no-pit-reinjection' : bug exists
without '-kvm-no-pit-reinjection' : bug exists


host kernel: 2.6.18-194.el5
# rpm -qa |grep kvm
kvm-83-170.el5
etherboot-zroms-kvm-5.4.4-13.el5
kvm-qemu-img-83-170.el5
kvm-debuginfo-83-170.el5
kmod-kvm-83-170.el5
kvm-tools-83-170.el5

(In reply to comment #17)
> (In reply to comment #16)
> > (In reply to comment #14)
> > > I also test on other machines, it's easy to reproduce when guest's memory size
> > > is same as host's.
> > 
> > That's almost certainly why the bug happens; likely that we are hitting swap on
> > the host and one of the qemu processes gets hosed.
> > 
> > It's also probable that this scenario is restricted to self migration.  Note
> > that self migration requires
> > 
> > 1) reading all memory from old VM
> > 2) passing data over network socket
> > 3) writing all memory to new VM
> > 
> > this strides over the entire host memory and requires easily > 2X the host
> > memory plus resources for network and processes.  You will swap, and
> > performance will be bad.
> 
> yes, the performance is very bad.

Can you make sure the host does not swap? We know for a fact that w/o pvclock time keeping is not 100% reliable. If the host is loaded or swap the guests it will greatly influence time keeping

(In reply to comment #18)

> Can you make sure the host does not swap? We know for a fact that w/o pvclock
> time keeping is not 100% reliable. If the host is loaded or swap the guests it
> will greatly influence time keeping    

This scenario is nearly 100% guarantee that it will introduce swap, not just small swap, but heavy, heavy swapping.  I don't think it's necessarily a migration or timekeeping bug; however, swapping during migration could be particularly bad for timekeeping.

There is possibly some hints we could provide to the mm layer using madvise; the access pattern here is going to run over 4G over memory with MADV_SEQUENTIAL semantics.

However, a 4 gigabyte memory self-migration on a 4 gigabyte host is not going to be a good performance scenario in any case, and isn't a realistic usage pattern for a real deployment.  If the problem won't exist on a host->host migration and won't exist if the memory sizes are within the capacity of the host, I suggest it's not a high priority bug.

I will still try an experiment with inducing swapping on the host to see if the effect is the same, perhaps we can do better catchup of timekeeping after, but it isn't likely we can address this at all in RHEL 5.

(In reply to comment #19)
> (In reply to comment #18)
> 
> > Can you make sure the host does not swap? We know for a fact that w/o pvclock
> > time keeping is not 100% reliable. If the host is loaded or swap the guests it
> > will greatly influence time keeping    
> 
> This scenario is nearly 100% guarantee that it will introduce swap, not just
> small swap, but heavy, heavy swapping.  I don't think it's necessarily a
> migration or timekeeping bug; however, swapping during migration could be
> particularly bad for timekeeping.


Yes, this bug only can be reproduce when swap is heavy. If set guest_mem to host_mem/2, the drift is very small.

I've tested the migration between two host, the drift is very small(0.28, this can be ignored). The memory size of guest and host are all 4G. 


> There is possibly some hints we could provide to the mm layer using madvise;
> the access pattern here is going to run over 4G over memory with
> MADV_SEQUENTIAL semantics.
> 
> However, a 4 gigabyte memory self-migration on a 4 gigabyte host is not going
> to be a good performance scenario in any case, and isn't a realistic usage
> pattern for a real deployment.  If the problem won't exist on a host->host
> migration and won't exist if the memory sizes are within the capacity of the
> host, I suggest it's not a high priority bug.
> 
> I will still try an experiment with inducing swapping on the host to see if the
> effect is the same, perhaps we can do better catchup of timekeeping after, but
> it isn't likely we can address this at all in RHEL 5.

In this case I'll close the bug as won't fix