Bug 953097

Summary: virtio-rng, boot the guest with two rng device, cat /dev/hwrng in guest, guest will call trace
Product: Red Hat Enterprise Linux 6 Reporter: Amit Shah <amit.shah>
Component: kernelAssignee: Amit Shah <amit.shah>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.4CC: acathrow, amit.shah, flang, juzhang, lnovich, mazhang, michen, qzhang, rhod, sradvan, virt-maint, yunzheng
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: kernel-2.6.32-375.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 917505 Environment:
Last Closed: 2013-11-21 18:14:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 917505    
Bug Blocks: 917953    
Attachments:
Description Flags
guest call trace vmcore-dmesg none

Description Amit Shah 2013-04-17 11:19:51 UTC 
RHEL6 clone

+++ This bug was initially created as a clone of Bug #917505 +++

Description of problem:
when i boot the guest (rhel6.4) with two rng device. the guest can boot normally.
but when i cat /dev/hwrng in guest. the guest will call trace

How reproducible:
100%

Steps to Reproduce:
1.boot guest with two rng device like :
 -device virtio-rng-pci \
 -device virtio-rng-pci
2. in guest  open /dev/hwrng
  # cat /dev/hwrng
3.
  
Actual results:
the guest will call trace and reboot.

Expected results:
the guest should work normally 

Additional info:
call trace info:
BUG: unable to handle kernel paging request at ffff8802a638b1fc
IP: [<ffffffffa005c65e>] virtqueue_add_buf+0x13e/0x480 [virtio_ring]
PGD 1a86063 PUD 0 
Oops: 0002 [#1] SMP 
last sysfs file: /sys/devices/pci0000:00/0000:00:05.0/usb1/speed
CPU 0 
Modules linked in: sunrpc ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 uinput microcode virtio_rng virtio_console virtio_net i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]

Pid: 1885, comm: cat Not tainted 2.6.32-355.el6.x86_64 #1 Bochs Bochs
RIP: 0010:[<ffffffffa005c65e>]  [<ffffffffa005c65e>] virtqueue_add_buf+0x13e/0x480 [virtio_ring]
RSP: 0018:ffff88011cbd9dc8  EFLAGS: 00010202
RAX: ffff880118d62870 RBX: ffff880118d62840 RCX: 0000000000000000
RDX: 000000018d628980 RSI: 0000000018d62898 RDI: 6db6db6db6db6db7
RBP: ffff88011cbd9e28 R08: ffff88011c2a92c0 R09: 0000160000000000
R10: ffff88011cbd9e58 R11: 0000000000000246 R12: ffff88011cbd9e38
R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000014
FS:  00007f8546a75700(0000) GS:ffff880028200000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: ffff8802a638b1fc CR3: 000000011cecf000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process cat (pid: 1885, threadinfo ffff88011cbd8000, task ffff88011c236aa0)
Stack:
 ffff88011cbd9ee8 ffffffff81510045 0000000000000000 ffffffff81286290
<d> ffff88011cbd9e38 ffff88011c2a92c4 ffff88011cbd9e28 ffff88011cbd9e38
<d> ffff88011c0583c0 0000000000007fec 0000000001e2f014 0000000000000014
Call Trace:
 [<ffffffff81510045>] ? page_fault+0x25/0x30
 [<ffffffff81286290>] ? sg_init_table+0x30/0x50
 [<ffffffffa002f053>] register_buffer+0x53/0x70 [virtio_rng]
 [<ffffffffa002f187>] virtio_data_read+0x57/0x70 [virtio_rng]
 [<ffffffff8134118e>] rng_dev_read+0x9e/0x1b0
 [<ffffffff811815f5>] vfs_read+0xb5/0x1a0
 [<ffffffff81181731>] sys_read+0x51/0x90
 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b


qemu-command:
/usr/libexec/qemu-kvm \
-name 'vm1' \
-nodefaults \
-m 4096 \
-smp 4,cores=2,threads=1,sockets=2 \
-vnc :22 \
-vga qxl \
-rtc base=utc,clock=host,driftfix=none \
-drive file=/root/RHEL-Server-6.4-64-virtio.qcow2,if=none,cache=none,id=virtio0 \
-device virtio-blk-pci,drive=virtio0 \
-device virtio-net-pci,netdev=id3Ibo2c,mac=9a:5e:5f:60:61:62 \
-netdev tap,id=id3Ibo2c,script=/root/qemu-ifup-switch \
-device ich9-usb-uhci1,id=usb1 \
-boot order=cdn,once=c,menu=on \
-enable-kvm \
-monitor stdio \
-global PIIX4_PM.disable_s3=0 \
-global PIIX4_PM.disable_s4=0 \
-chardev socket,id=qmp_id_qmpmonitor1,path=/tmp/qmpmonitor-1,server,nowait \
-mon chardev=qmp_id_qmpmonitor1,mode=control \
-chardev socket,id=isa-serial-1,path=/tmp/isa-serial-1,server,nowait \
-device isa-serial,chardev=isa-serial-1 \
-device virtio-serial,id=virt-serial-1,max_ports=31,bus=pci.0 \
-chardev socket,id=virtio-serial-1-1,path=/tmp/virtio-serial-1-1,server,nowait \
-device virtserialport,chardev=virtio-serial-1-1,name=virtio.serial.1.1,bus=virt-serial-1.0,id=virtio-serial-port1-1 \
-device virtio-serial,id=virt-console-1 \
-chardev socket,id=virtio-console-1-1,path=/tmp/virtio-console-1-1,server,nowait \
-device virtconsole,chardev=virtio-console-1-1,name=virtio.console.1.1,bus=virt-console-1.0 \
-device virtio-rng-pci \
-device virtio-rng-pci

host cpuinfo:
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                8
On-line CPU(s) list:   0-7
Thread(s) per core:    2
Core(s) per socket:    4
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 42
Stepping:              7
CPU MHz:               1600.000
BogoMIPS:              6784.61
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              8192K
NUMA node0 CPU(s):     0-7

--- Additional comment from langfang on 2013-03-26 08:16:10 IST ---

Seem hit the same problem: hook up the virtio-rng hwrng to the guest's entropy pool by running rngd:

Guest:
kernel-2.6.32-367.el6.x86_64

Steps:
1.Boot guest with 
..-object rng-random,id=rng0,filename=/dev/zero -device virtio-rng-pci,rng=rng0,bus=pci.0,addr=0x7 -object rng-random,id=rng1,filename=/dev/zero -device virtio-rng-pci,rng=rng1,bus=pci.0,addr=0x8..
2.In guest:

#cat /sys/devices/virtual/misc/hw_random/rng_available
virtio
#cat /sys/devices/virtual/misc/hw_random/rng_current
virtio
# rngd -r /dev/hwrng


Results: Call trace
<4>------------[ cut here ]------------
<2>kernel BUG at drivers/char/hw_random/virtio-rng.c:55!
<4>invalid opcode: 0000 [#1] SMP 
<4>last sysfs file: /sys/devices/pci0000:00/0000:00:08.0/irq
<4>CPU 0 
<4>Modules linked in: nls_utf8 fuse autofs4 sunrpc 8021q garp stp llc ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 microcode sg virtio_rng virtio_balloon e1000 i2c_piix4 i2c_core ext4 mbcache jbd2 sd_mod crc_t10dif sr_mod cdrom virtio_pci virtio_ring virtio ahci pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]
<4>
<4>Pid: 5874, comm: rngd Not tainted 2.6.32-367.el6.x86_64 #1 Bochs Bochs
<4>RIP: 0010:[<ffffffffa0019057>]  [<ffffffffa0019057>] register_buffer+0x57/0x70 [virtio_rng]
<4>RSP: 0018:ffff880119ef1e38  EFLAGS: 00010286
<4>RAX: 00000000ffffffe4 RBX: ffff880119ef1e38 RCX: 0000000000000001
<4>RDX: 0000000000000000 RSI: 0000000019136b48 RDI: ffff8801189f9180
<4>RBP: ffff880119ef1e68 R08: ffff8801188d6e80 R09: ffff880119ef1e38
<4>R10: 00007fff8fe6f410 R11: 0000000000000246 R12: ffff88011ae54bc0
<4>R13: 0000000000000990 R14: 00007fff8fe6ecb4 R15: 0000000000000034
<4>FS:  00007f2270a2d700(0000) GS:ffff880028200000(0000) knlGS:0000000000000000
<4>CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
<4>CR2: 00007fff8fe6ec68 CR3: 000000011c430000 CR4: 00000000000006f0
<4>DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<4>DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<4>Process rngd (pid: 5874, threadinfo ffff880119ef0000, task ffff880119cdaaa0)
<4>Stack:
<4> ffffea0003d5eed2 0000004000000e80 0000000000000000 0000000000000000
<4><d> ffff880119ef1e78 ffff880119ef0000 ffff880119ef1e78 ffffffffa0019187
<4><d> ffff880119ef1ee8 ffffffff8134164e 0000000000000004 ffff880100000001
<4>Call Trace:
<4> [<ffffffffa0019187>] virtio_data_read+0x57/0x70 [virtio_rng]
<4> [<ffffffff8134164e>] rng_dev_read+0x9e/0x1b0
<4> [<ffffffff811817a5>] vfs_read+0xb5/0x1a0
<4> [<ffffffff811818e1>] sys_read+0x51/0x90
<4> [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
<4>Code: 06 00 00 29 c2 e8 5a d9 26 e1 4c 8b 05 db 06 00 00 48 8b 3d cc 06 00 00 31 d2 b9 01 00 00 00 48 89 de e8 cd 24 00 00 85 c0 79 04 <0f> 0b eb fe 48 8b 3d ae 06 00 00 e8 19 20 00 00 48 83 c4 28 5b 
<1>RIP  [<ffffffffa0019057>] register_buffer+0x57/0x70 [virtio_rng]
<4> RSP <ffff880119ef1e38>
[root@localhost home]# 


addtional info :
1.CLI:
 /usr/libexec/qemu-kvm -enable-kvm -M pc-i440fx-1.4 -m 4G -smp 2 -uuid `uuidgen` -nodefaults -rtc base=utc  -device ahci,id=ahci0 -drive if=none,file=/mnt/rhel6.4-new.raw,format=raw,id=drive-sata0-0-0 -device ide-drive,bus=ahci0.0,drive=drive-sata0-0-0,id=sata0-0-0,bootindex=0  -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device e1000,netdev=hostnet0,id=net0,mac=22:14:08:60:3f:29,addr=0x4 -monitor stdio  -usb -device usb-tablet,id=input0  -qmp tcp:0:4445,server,nowait -spice port=5930,disable-ticketing  -vga qxl -serial unix:/tmp/tty0,server,nowait -device virtio-balloon-pci,id=balloon0,addr=0x6  -drive file=/home/RHEL6.4-20130123.0-Server-x86_64-DVD1.iso,if=none,media=cdrom,id=drive-ide0-1-0,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -object rng-random,id=rng0,filename=/dev/zero -device virtio-rng-pci,rng=rng0,bus=pci.0,addr=0x7 -object rng-random,id=rng1,filename=/dev/zero -device virtio-rng-pci,rng=rng1,bus=pci.0,addr=0x8
2.Please see attachment for the call trace file.

--- Additional comment from langfang on 2013-03-26 08:19:22 IST ---

Created attachment 716297 [details]
call trace

--- Additional comment from Amit Shah on 2013-04-17 16:43:02 IST ---

Fixed upstream by


commit e84e7a56a3aa2963db506299e29a5f3f09377f9b
Author: Amit Shah
Date:   Fri Mar 8 11:30:18 2013 +1100

    virtio: rng: disallow multiple device registrations, fixes crashes
    
    The code currently only supports one virtio-rng device at a time.
    Invoking guests with multiple devices causes the guest to blow up.
    
    Check if we've already registered and initialised the driver.  Also
    cleanup in case of registration errors or hot-unplug so that a new
    device can be used.
Comment 1 RHEL Program Management 2013-04-18 10:40:45 UTC 
This request was evaluated by Red Hat Product Management for
inclusion in a Red Hat Enterprise Linux release.  Product
Management has requested further review of this request by
Red Hat Engineering, for potential inclusion in a Red Hat
Enterprise Linux release for currently deployed products.
This request is not yet committed for inclusion in a release.
Comment 2 Jarod Wilson 2013-05-08 20:35:56 UTC 
Patch(es) available on kernel-2.6.32-375.el6
Comment 8 mazhang 2013-07-02 07:08:14 UTC 
Hi Amit,

I try verify this bug on qemu-kvm-0.12.1.2-2.376.el6.x86_64, but did not found virtio-rng-pci in qemu device list, so do we support this device in rhel6u4 ?

[root@localhost qemu]# uname -r
2.6.32-394.el6.x86_64
[root@localhost qemu]# rpm -qa |grep qemu
qemu-kvm-debuginfo-0.12.1.2-2.376.el6.x86_64
qemu-guest-agent-0.12.1.2-2.375.el6.x86_64
qemu-img-0.12.1.2-2.376.el6.x86_64
qemu-kvm-0.12.1.2-2.376.el6.x86_64
qemu-kvm-tools-0.12.1.2-2.376.el6.x86_64
gpxe-roms-qemu-0.9.7-6.9.el6.noarch
[root@localhost qemu]# /usr/libexec/qemu-kvm -device ?
name "pci-bridge", bus PCI
name "virtio-scsi-pci", bus PCI, alias "virtio-scsi"
name "virtio-balloon-pci", bus PCI
name "virtio-serial-pci", bus PCI, alias "virtio-serial"
name "virtio-net-pci", bus PCI
name "virtio-blk-pci", bus PCI, alias "virtio-blk"
name "i82562", bus PCI
name "i82559er", bus PCI
name "i82559c", bus PCI
name "i82559b", bus PCI
name "i82559a", bus PCI
name "i82558b", bus PCI
name "i82558a", bus PCI
name "i82557c", bus PCI
name "i82557b", bus PCI
name "i82557a", bus PCI
name "i82551", bus PCI
name "i82550", bus PCI
name "pcnet", bus PCI
name "rtl8139", bus PCI
name "e1000", bus PCI, desc "Intel Gigabit Ethernet"
name "smbus-eeprom", bus I2C
name "usb-hub", bus USB
name "usb-host", bus USB
name "usb-kbd", bus USB
name "usb-mouse", bus USB
name "usb-tablet", bus USB
name "usb-storage", bus USB
name "scsi-disk", bus SCSI, desc "virtual SCSI disk or CD-ROM (legacy)"
name "scsi-block", bus SCSI, desc "SCSI block device passthrough"
name "scsi-cd", bus SCSI, desc "virtual SCSI CD-ROM"
name "scsi-hd", bus SCSI, desc "virtual SCSI disk"
name "scsi-generic", bus SCSI, desc "pass through generic scsi device (/dev/sg*)"
name "usb-ccid", bus USB, desc "CCID Rev 1.1 smartcard reader"
name "ccid-card-passthru", bus ccid-bus, desc "passthrough smartcard"
name "usb-redir", bus USB
name "virtserialport", bus virtio-serial-bus
name "virtconsole", bus virtio-serial-bus
name "ich9-usb-ehci1", bus PCI
name "usb-ehci", bus PCI
name "i6300esb", bus PCI
name "ne2k_pci", bus PCI
name "ide-drive", bus IDE
name "isa-ide", bus ISA
name "ES1370", bus PCI, desc "ENSONIQ AudioPCI ES1370"
name "AC97", bus PCI, desc "Intel 82801AA AC97 Audio"
name "intel-hda", bus PCI, desc "Intel HD Audio Controller"
name "hda-micro", bus HDA, desc "HDA Audio Codec, duplex (speaker, microphone)"
name "hda-duplex", bus HDA, desc "HDA Audio Codec, duplex (line-out, line-in)"
name "hda-output", bus HDA, desc "HDA Audio Codec, output-only (line-out)"
name "VGA", bus PCI
name "SUNW,fdtwo", bus System
name "sysbus-fdc", bus System
name "isa-serial", bus ISA
name "cirrus-vga", bus PCI, desc "Cirrus CLGD 54xx VGA"
name "isa-parallel", bus ISA
name "sga", bus ISA, desc "Serial Graphics Adapter"
name "ich9-usb-uhci3", bus PCI
name "ich9-usb-uhci2", bus PCI
name "ich9-usb-uhci1", bus PCI
name "piix4-usb-uhci", bus PCI
name "piix3-usb-uhci", bus PCI
name "ib700", bus ISA
name "ne2k_isa", bus ISA
name "testdev", bus ISA
name "isa-debugcon", bus ISA
name "pci-assign", bus PCI, desc "pass through host pci devices to the guest"
name "qxl", bus PCI, desc "Spice QXL GPU (secondary)"
name "qxl-vga", bus PCI, desc "Spice QXL GPU (primary, vga compatible)"


Thanks,
Mazhang.
Comment 9 mazhang 2013-07-02 08:03:23 UTC 
reproduce on rhel6u4 guest kernel-2.6.32-358.el6.x86_64.

Steps to Reproduce:
1.boot guest with two rng device:
/usr/libexec/qemu-kvm \
-M pc \
-cpu SandyBridge \
-m 2G \
-smp 2,sockets=1,cores=2,threads=1,maxcpus=16 \
-enable-kvm \
-name win7 \
-uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \
-smbios type=1,manufacturer='Red Hat',product='RHEV Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3-e63fcdb67170 \
-k en-us \
-rtc base=localtime,clock=host,driftfix=slew \
-no-kvm-pit-reinjection \
-monitor stdio \
-qmp tcp:0:6667,server,nowait \
-boot menu=on \
-bios /usr/share/seabios/bios.bin \
-netdev tap,id=hostnet0,downscript=no,vhost=on \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:2e:28:1c,bus=pci.0,addr=0x4 \
-vga qxl \
-spice port=5901,disable-ticketing \
-global PIIX4_PM.disable_s3=0 \
-global PIIX4_PM.disable_s4=0 \
-drive file=/home/rhel6u5.qcow2,if=none,id=drive-ide0-0-0,format=qcow2,cache=none,aio=threads \
-device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 \
-device virtio-rng-pci \
-device virtio-rng-pci \

2. in guest  open /dev/hwrng
  # cat /dev/hwrng
  
Results:
the guest will call trace and reboot.
Comment 10 mazhang 2013-07-02 08:04:32 UTC 
Created attachment 767662 [details]
guest call trace vmcore-dmesg
Comment 11 mazhang 2013-07-02 08:15:14 UTC 
verify this bug on rhel6u4 guest kernel-2.6.32-392.el6.x86_64

Steps to Reproduce:
1.boot guest with two rng device:
/usr/libexec/qemu-kvm \
-M pc \
-cpu SandyBridge \
-m 2G \
-smp 2,sockets=1,cores=2,threads=1,maxcpus=16 \
-enable-kvm \
-name win7 \
-uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \
-smbios type=1,manufacturer='Red Hat',product='RHEV Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3-e63fcdb67170 \
-k en-us \
-rtc base=localtime,clock=host,driftfix=slew \
-no-kvm-pit-reinjection \
-monitor stdio \
-qmp tcp:0:6667,server,nowait \
-boot menu=on \
-bios /usr/share/seabios/bios.bin \
-netdev tap,id=hostnet0,downscript=no,vhost=on \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:2e:28:1c,bus=pci.0,addr=0x4 \
-vga qxl \
-spice port=5901,disable-ticketing \
-global PIIX4_PM.disable_s3=0 \
-global PIIX4_PM.disable_s4=0 \
-drive file=/home/rhel6u5.qcow2,if=none,id=drive-ide0-0-0,format=qcow2,cache=none,aio=threads \
-device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 \
-device virtio-rng-pci \
-device virtio-rng-pci \

2. in guest  open /dev/hwrng
  # cat /dev/hwrng
  
Results:
guest will not call trace.
Comment 14 errata-xmlrpc 2013-11-21 18:14:18 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-1645.html