Bug 1481701

> 
> ipxe-roms-qemu-20160127-5.git6366fa7a.el7.noarch            Tue Nov 29
> 02:23:11 2016
> libvirt-daemon-driver-qemu-2.0.0-10.el7_3.9.x86_64          Wed Jul 19
> 20:44:33 2017
> qemu-img-rhev-2.6.0-28.el7_3.9.x86_64                       Wed Jul 19
> 20:44:32 2017
> qemu-kvm-common-rhev-2.6.0-28.el7_3.9.x86_64                Wed Jul 19
> 20:44:18 2017
> qemu-kvm-rhev-2.6.0-28.el7_3.9.x86_64                       Wed Jul 19
Update the component to qemu-kvm-rhev according to above log.

Hi Pankaj,

QE have never hit this issue before with our test cases, and I can't figure out the reproduce steps from comment 0 and comment 9, which do not clearly show the scenario before the crash. This is no even qemu command line or virsh xml provided. It is really hard for QE to reproduce it. 

The only valuable information I can see is the assertion:
qemu-kvm: hw/char/serial.c:232: serial_xmit: Assertion `!(s->lsr & 0x40)' failed.

Is that possible by reviewing the code of hw/char/serial.c to find out the possible path to trigger the issue?

Thanks,
Qianqian

Hi Gyanendra Kumar,

Would you help provide the qemu commmand line or virsh xml of the guest, and any application running or stress load on guest? And are there any other actions like reboot, snapshot or migration?

Thanks,
Qianqian

(In reply to Qianqian Zhu from comment #12)
> Hi Gyanendra Kumar,
> 
> Would you help provide the qemu commmand line or virsh xml of the guest, and
> any application running or stress load on guest? And are there any other
> actions like reboot, snapshot or migration?
> 
> Thanks,
> Qianqian

Sorry, I just found the qemu command line and some guest actions in the sosreport. I will try to reproduce it. Thanks.

QE can not reproduce it so far, by trying below scenarios:

1. On rhel7.3 host, boot up 5 rhel7.3 guests at the same time. 2 guests running on remote rbd storage, and 3 guests running on local image.
2. Keep them running for 20 days, continuously loading each guest with netperf and fio.
3. Guest's configuration for char device:
    <serial type='file'>
      <source path='/var/lib/libvirt/95aa309e-d4f9-4967-b5cb-d5d67aa6335d/console.log'/>
      <target port='1'/>
      <alias name='serial0'/>
    </serial>
    <serial type='pty'>
      <source path='/dev/pts/5'/>
      <target port='2'/>
      <alias name='serial1'/>
    </serial>
    <console type='file'>
      <source path='/var/lib/libvirt/95aa309e-d4f9-4967-b5cb-d5d67aa6335d/console.log'/>
      <target type='serial' port='1'/>
      <alias name='serial0'/>
    </console>

qemu command line:
/usr/libexec/qemu-kvm \
-name guest=instance-00000191,debug-threads=on \
-S \
-object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-instance-00000191/master-key.aes \
-machine pc-i440fx-rhel7.4.0,accel=kvm,usb=off,dump-guest-core=off \
-cpu SandyBridge,vme=on,ss=on,pcid=on,hypervisor=on,arat=on,tsc_adjust=on,pdpe1gb=on,xsave=off,avx=off \
-m 8192 -realtime mlock=off \
-smp 8,sockets=8,cores=1,threads=1 \
-uuid 95aa309e-d4f9-4967-b5cb-d5d67aa6335d \
-smbios type=1,manufacturer=Red Hat,product=OpenStack Nova,version=13.1.1-7.el7ost,serial=6d20bf45-0918-413b-b4bf-dcc702889837,uuid=95aa309e-d4f9-4967-b5cb-d5d67aa6335d,family=Virtual Machine \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-1-instance-00000191/monitor.sock,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-boot strict=on \
-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 \
-object secret,id=virtio-disk0-secret0,data=VwmHY95rvwrjGG/R/NeLGD+SMxn2TA23xNy8cGFFVyo=,keyid=masterKey0,iv=6CNUM2pL3uqMD0kKUNEGuQ==,format=base64 \
-drive file=rbd:libvirt-pool/vol2:id=libvirt:auth_supported=cephx\;none:mon_host=x.x.x.x,file.password-secret=virtio-disk0-secret0,format=raw,if=none,id=drive-virtio-disk0,cache=writeback,discard=unmap -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \
-object secret,id=virtio-disk1-secret0,data=1Q/qBPk1f4hf59M1ZmIKgfpcLBNBV2lM9HbYMlMGcks=,keyid=masterKey0,iv=0nmgrk9+Y2J0NDrpp1oGyA==,format=base64 \
-drive file=rbd:libvirt-pool/vol1:id=libvirt:auth_supported=cephx\;none:mon_host=x.x.x.x,file.password-secret=virtio-disk1-secret0,format=raw,if=none,id=drive-virtio-disk1,cache=writeback \
-device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk1,id=virtio-disk1 -netdev tap,fd=25,id=hostnet0,vhost=on,vhostfd=27 \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:2c:3f:66,bus=pci.0,addr=0x3 -netdev tap,fd=28,id=hostnet1,vhost=on,vhostfd=29 \
-device virtio-net-pci,netdev=hostnet1,id=net1,mac=fa:16:3e:e0:c3:c2,bus=pci.0,addr=0x4 \
-add-fd set=4,fd=31 \
-chardev file,id=charserial0,path=/dev/fdset/4,append=on \
-device isa-serial,chardev=charserial0,id=serial0 \
-chardev pty,id=charserial1 \
-device isa-serial,chardev=charserial1,id=serial1 \
-device usb-tablet,id=input0,bus=usb.0,port=1 \
-vnc 0.0.0.0:0 \
-k en-us \
-device cirrus-vga,id=video0,bus=pci.0,addr=0x2 \
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 \
-msg timestamp=on

Hi,

Below recent commits in >= qemu-kvm-rhev-2.12.0-8.el7 releases. Both these commits fix potential issues with UART serial code emulation. As we don't have
exact reproduction steps from any of the attached cases and we need help from Customer/GSS to confirm if above mentioned qemu-kvm-rhev solves the problem?

1] d357e5f3a56416c6f5a7a9308b8c30f825d89cfa
2] a7a6995e2a24e9e4c89084acf3241e5640f30f3e

More Details:
----------------
Code inspection tells both commit 1] & commit 2] fixes a condition when retry logic does not execute in case of data transmit failure or connection hungup.

- If 's->lsr & UART_LSR_THRE' is set for either FIFO mode(no data to transmit)     
  or regular byte transfer mode.

- If data transfer(qemu_chr_fe_write) fails due to some reason. It does not  
  execute retry logic.   

- This results to exit main 'while loop' as UART_LSR_THRE is set.
  "(while (!(s->lsr &  UART_LSR_THRE));)" 

- This sets Line state register as transmitter empty.
      s->lsr |= UART_LSR_TEMT;

- If any of callback is still active, there is possibility of asserting on                    "assert(!(s->lsr & UART_LSR_TEMT));"

Based on above justification (only by code inspection), I feel we can ask Customer to test qemu-kvm-rhev-2.12 and based on results can back-port these commits in older versions if required.

Also, it would be good if we get some steps to reproduce issue locally to confirm the analysis. 

Thanks,
Pankaj

Moving to verified per above comments.

Hi Redhat Team,

We are in a process of upgrading the RHEL 7.3 to 7.5. But we see that 'qemu-kvm-rhev-2.12.0-8.el7' is not available in the 7.5 release but a lower version. What may be the solution to have the fix for this in RHEL 7.5.
Thanks
Nitesh

Hi Nitesh,

(In reply to Nitesh from comment #35)
> Hi Redhat Team,
> 
> We are in a process of upgrading the RHEL 7.3 to 7.5. But we see that
> 'qemu-kvm-rhev-2.12.0-8.el7' is not available in the 7.5 release but a lower
> version. What may be the solution to have the fix for this in RHEL 7.5.
> Thanks
> Nitesh

We will be working on a backport to RHEL 7.5.z as well for this one. Once the clone is made, you can follow the progress in the cloned BZ for RHEL 7.5.z

Hi,

I can see the clone , but there is no progress on the clone.

Thanks
Nitesh