Bug 1657095

Correct info in step 16:
  start guest the full backed-up image "back1.img": --> start guest with the incremental backed-up image "back2.img":

This workflow looks correct to me.

Eric, can you take a glance at just the sequence of commands here and let me know if you see something fishy? I think this ought to work as written; but I did not delve into the contents of copyif2.sh to know if something there could be the culprit.

Also hit this issue in qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64.

(In reply to aihua liang from comment #0)
> Created attachment 1512365 [details]
> Start_guest_failure
> 
> Description of problem:
>   Incremental live backup pull mode: Failed to start guest with the
> incremental backed-up image of system disk after its rebase
> 

> 
> 7. Client: create a image as full back-up image.
>     (host)# qemu-img create -f qcow2 back1.img 20G
> 
> 8. Client: pull full back-up info from server.
>     (host)# ./copyif2.sh nbd://10.73.196.67:10809/tmp back1.img

So far, seems okay - back1.img should be a backup of the guest (although without using qga to do a freeze, the backup may be in a slightly inconsistent state that would require an fsck to replay the journal to be usable).

Visually, we could imagine that back1.img contains AAAA---- corresponding to the point in time when the backup was started.

> 
> 9. Client: 
>      9.1 check image info back1.img
>        (host)# qemu-img info back1.img
>        ---> the disk size should be the same with that of step 5.
>      9.2 check if the full backed-up image can work well:
>        start guest the full backed-up image "back1.img":
>        ....
>        -object iothread,id=iothread0 \
>        -device
> virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x3,id=iothread0 \
>        -blockdev
> driver=qcow2,node-name=drive_image1,cache.no-flush=on,cache.direct=off,file.
> driver=file,file.aio=threads,file.filename=/home/back1.img \
>        -device scsi-hd,id=image1,drive=drive_image1 \

Wait - did you just start a new disk directly on back1.img being a writable image?  If back1.img is writable, the act of booting a guest with that image may have modified the image.  Any further incremental backup built on top of back1.img may have been corrupted, if back1.img is not left pristine.

Visually, if loading back1.img writable caused any journal replay, we may now have ABAAB--- in back1.img.

If you were to start your temporary qemu with -snapshot=on, that would leave back1.img pristine (by doing any write modifications due to journal reply in the transient overlay image, rather than directly in back1.img).


> 
> 14. Client: create a image as incremental backed-up image
>      (host)# qemu-img create -f qcow2 back2.img 20G
> 
> 15. Client: pull full back-up info from server.
>     (host)# ./copyif2.sh nbd://10.73.196.67:10809/tmp back2.img bitmap0

This should copy into back2.img just the clusters that were changed since the point in time at which back1.img was created.

Visually, if the guest was at ACCA-C-- at the time the backup was started, then back2.img contains just -CC--C--.


> 
> 16. Client: Start guest with the incremental backed-up image after rebase.
>      (host)# qemu-img rebase -u -f qcow2 -F qcow2 -b back1.img back2.img

and this says that back2.img should now be used as an overlay on top of back1.img.  If back1.img is still AAAA----, then viewing back2.img would see the correct ACCA-C--.  But if back1.img has been corrupted to ABAAB---, then viewing back2.img would see ACAABC--, which is NOT the same as what the point in time backup was supposed to see.

>      
>      start guest the full backed-up image "back1.img":
>        ....
>        -object iothread,id=iothread0 \
>        -device
> virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x3,id=iothread0 \
>        -blockdev
> driver=qcow2,node-name=drive_image1,cache.no-flush=on,cache.direct=off,file.
> driver=file,file.aio=threads,file.filename=/home/back2.img \
>        -device scsi-hd,id=image1,drive=drive_image1 \
>        ....
>  
>    ---> In client, guest failed to start with "back2.img" with error info
> "Failed to start Switch Root".

So this step is suspect if the earlier steps did not take care to preserve back1.img unaltered.

I also did my testing on a data disk separate from the OS disk, so that I had full control over what modifications where being made to the data disk between operations, without having to worry about things like qga for fsfreeze.  But using fsfreeze in this sequence may also help in creating a backup image that is less likely to need an fsck/journal reply on boot (the two points where it matters are the creation of the backup at step 4 and 11, so you'd want to play with inserting a freeze at step 3.5 and 10.5, and a thaw at step 4.5 and 11.5).


> Additional info:
>  Also hit this issue on RHEL7.6.z:
>    kernel version: 3.10.0-957.1.2.el7.x86_64
>    qemu-kvm-rhev version: qemu-kvm-rhev-2.12.0-18.el7_6.3.x86_64
> 
>  #cat copyif2.sh 
>  copyif2() {
...

> while read line; do
>   [[ $line =~ .*start.:.([0-9]*).*length.:.([0-9]*).*data.:.$state.* ]] ||
> continue
>   start=${BASH_REMATCH[1]} len=${BASH_REMATCH[2]}
>   echo
>   echo " $start $len:"
>   ####made some modification simply here, for qemu-io failed to execute with
> $len greater than 2147483136
>   if [ $len -gt 2147483136 ]
>       then
>           let left_length=$len-2147483136
>           let new_start=$start+2147483136
>           qemu-io -C -c "r $start 2147483136" -f qcow2 $2
>           qemu-io -C -c "r $new_start $left_length " -f qcow2 $2
>   else
>       qemu-io -C -c "r $start $len " -f qcow2 $2
>   fi

Yeah, I guess I never tested large images when I generated a shell script for use in testing backups.  This modification looks sane as long as the dirty bitmap never encodes more than 4G of image at once (because you've basically split it into a 2G prefix and the rest) - but if a dirty image has more than 4G of contiguous dirty clusters, this would need to be a while loop that iterates a 2G chunk at a time rather than just an if statement that hardcodes a single split.  I don't think it affects this current bug report, though.

Yes, Eric is right, if i check back1.img by booting it up during step9.2 &9.3, guest will failed to start after image rebase in step16; otherwise, it works ok.

 Retest it with following steps on qemu-img-4.1.0-13.module+el8.1.0+4313+ef76ec61.x86_64, it works ok.

 Test Steps:
   1.Server: start guest with qemu cmds
   /usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1'  \
    -sandbox off  \
    -machine pc  \
    -nodefaults \
    -device VGA,bus=pci.0,addr=0x2  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20181128-054651-YPtCwugS,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20181128-054651-YPtCwugS,server,nowait \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=id9FildC  \
    -chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20181128-054651-YPtCwugS,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20181128-054651-YPtCwugS,path=/var/tmp/seabios-20181128-054651-YPtCwugS,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20181128-054651-YPtCwugS,iobase=0x402 \
    -device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 \
    -device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 \
    -device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 \
    -device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 \
    -object iothread,id=iothread0 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x3,id=iothread0 \
    -blockdev driver=qcow2,node-name=drive_image1,cache.no-flush=on,cache.direct=off,file.driver=file,file.aio=threads,file.filename=/home/kvm_autotest_root/images/rhel80-64-virtio-scsi.qcow2 \
    -device scsi-hd,id=image1,drive=drive_image1 \
    -device virtio-net-pci,mac=9a:6f:70:71:72:73,id=idJzalak,vectors=4,netdev=idIEbgxj,bus=pci.0,addr=0x4  \
    -netdev tap,id=idIEbgxj,vhost=on \
    -m 4096  \
    -smp 10,maxcpus=10,cores=5,threads=1,sockets=2  \
    -cpu 'Broadwell',+kvm_pv_unhalt \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot order=cdn,once=d,menu=off,strict=off  \
    -enable-kvm \
    -monitor stdio \
    -qmp tcp:0:3000,server,nowait \
    -blockdev driver=qcow2,node-name=drive_data1,cache.no-flush=on,cache.direct=off,file.driver=file,file.aio=threads,file.filename=/home/data1 \
    -device scsi-hd,id=data1,drive=drive_data1 \
    -blockdev driver=qcow2,node-name=drive_data2,cache.no-flush=on,cache.direct=off,file.driver=file,file.aio=threads,file.filename=/home/data2 \
    -device scsi-hd,id=data2,drive=drive_data2 \
    -blockdev driver=qcow2,node-name=drive_data3,cache.no-flush=on,cache.direct=off,file.driver=file,file.aio=threads,file.filename=/home/data3 \
    -device scsi-hd,id=data3,drive=drive_data3 \
    -spice ipv4,disable-ticketing,port=4000 \

2. Server: start nbd server 
    {"execute":"nbd-server-start","arguments":{"addr":{"type":"inet","data":{"host":"10.73.224.68","port":"10809"}}}}

*********full backup of system disk*************
3. Server: in guest, create a file and record its md5sum.
    (guest)# touch a
    (guest)# md5sum ? | tee sum1

4. Server: start a channel for full backup
    (host)# qemu-img create -f qcow2 -b /home/kvm_autotest_root/images/rhel80-64-virtio-scsi.qcow2 -F qcow2 scratch.img
    {"execute":"blockdev-add","arguments":{"driver":"qcow2","node-name":"tmp","file":{"driver":"file","filename":"/home/scratch.img"},"backing":"drive_image1"}}
    { "execute": "transaction", "arguments": { "actions": [ {"type": "blockdev-backup", "data": { "device": "drive_image1", "target": "tmp", "sync": "none", "job-id":"j1" } }, {"type": "block-dirty-bitmap-add", "data": { "node": "drive_image1", "name": "bitmap0" } } ] } }
    {"execute":"nbd-server-add","arguments":{"device":"tmp"}}

5. Server: check the image info of system disk
    (host)# qemu-img info home/kvm_autotest_root/images/rhel80-64-virtio-scsi.qcow2 -U

6. Server: in guest, add another new file and record its md5sum
    (guest)# dd if=/dev/zero of=b bs=1M count=1
    (guest)# md5sum ? | tee sum2

7. Client: create a image as full back-up image.
    (host)# qemu-img create -f qcow2 back1.img 20G

8. Client: pull full back-up info from server.
    (host)# ./copyif2.sh nbd://10.73.224.68:10809/tmp back1.img

9. Client: check image info back1.img
       (host)# qemu-img info back1.img
       ---> the disk size should be the same with that of step 5.
10. Server: clean up env after full back-up.
   qmp:
    {"execute":"nbd-server-remove","arguments":{"name":"tmp"}}
    {"execute":"block-job-cancel","arguments":{"device":"j1"}}
    {"execute":"blockdev-del","arguments":{"node-name":"tmp"}}
   host:
    #rm scratch.img
    


*****************incremental backup of system disk*********

11. Server: start a channel for incremental backup
    (host)#qemu-img create -f qcow2 -b /home/kvm_autotest_root/images/rhel80-64-virtio-scsi.qcow2 -F qcow2 scratch.img
    {"execute":"blockdev-add","arguments":{"driver":"qcow2","node-name":"tmp","file":{"driver":"file","filename":"/home/scratch.img"},"backing":"drive_image1"}}
    { "execute": "transaction", "arguments": { "actions": [ {"type": "blockdev-backup", "data": { "device": "drive_image1", "target": "tmp", "sync": "none","job-id":"j1" } }, {"type": "block-dirty-bitmap-add", "data": { "node": "drive_image1", "name": "bitmap1" }}, {"type": "block-dirty-bitmap-disable", "data": { "node": "drive_image1", "name": "bitmap0" } } ] } }
   {"execute":"nbd-server-add","arguments":{"device":"tmp","bitmap":"bitmap0"}}

12. Server: check image info of system disk
   (host)# qemu-img info home/kvm_autotest_root/images/rhel80-64-virtio-scsi.qcow2 -U

13. Server: in guest, add another new file and record its md5sum
    (guest)# dd if=/dev/urandom of=c bs=1M count=10    
    (guest)# md5sum ? | tee sum3

14. Client: create a image as incremental backed-up image
     (host)# qemu-img create -f qcow2 back2.img 20G

15. Client: pull full back-up info from server.
    (host)# ./copyif2.sh nbd://10.73.224.68:10809/tmp back2.img bitmap0

16. Client: Start guest with the incremental backed-up image after rebase.
     (host)# qemu-img rebase -u -f qcow2 -F qcow2 -b back1.img back2.img
     
     start guest the full backed-up image "back2.img":
       ....
       -object iothread,id=iothread0 \
       -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x3,id=iothread0 \
       -blockdev driver=qcow2,node-name=drive_image1,cache.no-flush=on,cache.direct=off,file.driver=file,file.aio=threads,file.filename=/home/back2.img \
       -device scsi-hd,id=image1,drive=drive_image1 \
       ....

After step16, guest can start up successfully.


Will update test steps of my case, thanks for John's and Eric's checking and analysing..

And Close this bug as a notabug.