Bug 928154

Summary: rmmod blocked when execute "rmmod virtio_rng" in rhel7 guest
Product: Red Hat Enterprise Linux 7 Reporter: langfang <flang>
Component: qemu-kvmAssignee: Amit Shah <amit.shah>
Status: CLOSED NOTABUG QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.0CC: acathrow, hhuang, juzhang, qzhang, rhod, sluo, virt-maint, xfu, yunzheng
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-03-29 10:46:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 917953    

Description langfang 2013-03-27 03:46:38 UTC
Description of problem:
 when execute "rmmod virtio_rng" in rhel7 guest,show error info:

[  600.950168] INFO: task rmmod:2057 blocked for more than 120 seconds.
[  600.953073] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this 

Version-Release number of selected component (if applicable):

Host:
# uname -r
3.9.0-0.rc3.44.el7.x86_64
# rpm -q qemu-kvm
qemu-kvm-1.4.0-1.el7.x86_64

Guest:
rhel7 
3.9.0-0.rc3.44.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Boot guest with
 ...-device virtio-rng-pci -device virtio-rng-pci ..

2.In guest
# lspci  |grep Un
00:03.0 Unclassified device [00ff]: Red Hat, Inc Device 1005
00:05.0 Unclassified device [00ff]: Red Hat, Inc Device 1005
00:06.0 Unclassified device [00ff]: Red Hat, Inc Virtio memory balloon

3.In guest 
#lsmod |grep rng
virtio_rng
#rmmod virtio_rng

Actual results:

ON  host:

Expected results:
asks: 169 total,   1 running, 168 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.6 us,  0.1 sy,  0.0 ni, 98.1 id,  0.1 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:   7848932 total,  5076920 used,  2772012 free,        8 buffers
KiB Swap:  8126460 total,        0 used,  8126460 free,  3368020 cached

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                
32197 root      20   0 6620672  92360   6160 S 100.4 1.177  31:22.31 qemu-kvm       

guest log:
[  OK  ] Started System Logging Service.
         Starting LSB: Bring up/down networking...
         Starting D-Bus System Message Bus...
[  OK  ] Started D-Bus System Message Bus.
[  OK  ] Started Restore Sound Card State.
[  OK  ] Started Permit User Sessions.
         Starting Command Scheduler...

Red Hat Enterprise Linux Server release 7.0 Alpha3 (Maipo)
Kernel 3.9.0-0.rc3.44.el7.x86_64 on an x86_64

dhcp-66-71-120 login: [  240.934254] INFO: task rmmod:2057 blocked for more than 120 seconds.
[  240.936680] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  360.940190] INFO: task rmmod:2057 blocked for more than 120 seconds.
[  360.942647] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  480.944157] INFO: task rmmod:2057 blocked for more than 120 seconds.
[  480.946690] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  600.950168] INFO: task rmmod:2057 blocked for more than 120 seconds.
[  600.953073] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  720.956191] INFO: task rmmod:2057 blocked for more than 120 seconds.
[  720.958809] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  840.962235] INFO: task rmmod:2057 blocked for more than 120 seconds.
[  840.964759] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  960.966178] INFO: task rmmod:2057 blocked for more than 120 seconds.
[  960.968165] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1080.970166] INFO: task rmmod:2057 blocked for more than 120 seconds.
[ 1080.972374] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1200.975138] INFO: task rmmod:2057 blocked for more than 120 seconds.

...
Additional info:

On  rhel6 guest ,not hit the problem.

Comment 1 langfang 2013-03-27 03:52:27 UTC
MY CLI:
/usr/libexec/qemu-kvm -enable-kvm -M q35 -m 4G -smp 2 -uuid ee025462-2997-404c-a220-69ef3f225756 -nodefaults -rtc base=utc -drive file=/home/rhel7-64-1.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,aio=native,media=disk,werror=stop,rerror=stop -device virtio-scsi-pci,id=bus1 -device scsi-hd,bus=bus1.0,drive=drive-virtio-disk0,id=virtio-scsi-pci0 -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:60:3f:29,addr=0x4 -monitor stdio -boot menu=on -qmp tcp:0:4445,server,nowait -vnc :10 -vga std -serial unix:/tmp/tty0,server,nowait -device virtio-balloon-pci,id=balloon0,addr=0x6 -acpitable file=/home/seabios/out/q35-acpi-dsdt.aml -drive file=/root/RHEL-7.0-20130306.0-Server-x86_64-dvd1.iso,if=none,media=cdrom,id=drive-ide0-1-0,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=0 -acpitable file=/home/seabios/out/q35-acpi-dsdt.aml -device virtio-rng-pci -device virtio-rng-pci

Comment 5 Hai Huang 2013-03-27 19:57:58 UTC
Not sure "rmmod virtio_rng" on the guest is a valid test case.
Please dispose appropriately.  Thanks.

Comment 6 Amit Shah 2013-03-28 08:41:19 UTC
Is rngd running in the guest?  That might block the module reomve.  If you kill rngd, does remove succeed?

Comment 7 Amit Shah 2013-03-28 09:08:39 UTC
Pls see comment 6 for the question

Comment 8 langfang 2013-03-29 03:08:14 UTC
(In reply to comment #6)
> Is rngd running in the guest?  That might block the module reomve.  If you
> kill rngd, does remove succeed?

HI,amit

   You are right, If i kill rngd,can remove the module "virtio_rng" successfully.Then if i reboot the guest,this module load automatically again.


addtional info:
 

 If i kill rngd in guest --> remove the module "virtio_rng" successfully--->reboot guest will the such log(see as following)--->load the virtio_rng module automatically  for the guest.


....
[  OK  ] Started Save Random Seed.
[  OK  ] Started Update UTMP about System Shutdown.
[  OK  ] Stopped Monitoring of LVM2 mirrors, snapshots etc. u...ogress polling.
         Stopping Initialize storage subsystems (RAID, LVM, etc.)...
[  OK  ] Stopped Initialize storage subsystems (RAID, LVM, etc.).
[  OK  ] Stopped target Encrypted Volumes.
         Stopping Initialize storage subsystems (RAID, LVM, etc.)...
[  OK  ] Stopped Initialize storage subsystems (RAID, LVM, etc.).
[  OK  ] Reached target Shutdown.
Sending SIGTERM to remaining processes...
Sending SIGKILL to remaining processes...
Unmounting file systems.
Unmounting /sys/fs/fuse/connections.
Unmounting /boot.
Unmounting /sys/kernel/config.
Unmounting /sys/kernel/debug.
Unmounting /dev/hugepages.
Unmounting /dev/mqueue.
Unmounting /var/lib/nfs/rpc_pipefs.
All filesystems unmounted.
Deactivating swaps.
All swaps deactivated.
Detaching loop devices.
Detaching loopback /dev/loop7.
Detaching loopback /dev/loop6.
Detaching loopback /dev/loop5.
Detaching loopback /dev/loop4.
Detaching loopback /dev/loop3.
Detaching loopback /dev/loop2.
Detaching loopback /dev/loop1.
Detaching loopback /dev/loop0.
All loop devices detached.
Detaching DM devices.
Detaching DM 253:2.
Detaching DM 253:0.
Not all DM devices detached, 1 left.
Detaching DM devices.
Not all DM devices detached, 1 left.
Cannot finalize remaining file systems and devices, giving up.
Successfully changed into root pivot.
dracut Warning: Killing all remaining processes
Rebooting.
[  126.854171] Restarting system.
[  OK  ] Started Show Plymouth Boot Screen.
[  OK  ] Reached target Basic System.

Welcome to Linux!

         Expecting device dev-ttyS0.device...
[  OK  ] Listening on Syslog Socket.
[  OK  ] Reached target Syslog.
[  OK  ] Reached target Remote File Systems.
         Starting Load legacy module configuration...
[  OK  ] Listening on Delayed Shutdown Socket.
[  OK  ] Listening on /dev/initctl Compatibility Named Pipe.
         Starting Setup Virtual Console...
         Starting Apply Kernel Variables...
         Starting Load Kernel Modules...
         Mounting Debug File System...
[  OK  ] Set up automount Arbitrary Executable File Formats F...utomount Point.
         Mounting POSIX Message Queue File System...
         Mounting Huge Pages File System...
[  OK  ] Reached target Encrypted Volumes.
....
....
[  OK  ] Started Command Scheduler.
         Starting Job spooling tools...

Red Hat Enterprise Linux Server release 7.0 Alpha3 (Maipo)
Kernel 3.9.0-0.rc3.44.el7.x86_64 on an x86_64

dhcp-66-71-120 login:

Comment 9 Amit Shah 2013-03-29 10:46:16 UTC
Yes, you can't remove modules while they're in use.  lsmod output shows whether the module is used.