Bug 730609
| Summary: | Guest can not init virtio serial port when do hotplug virtio serial frequently | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 6 | Reporter: | Joy Pu <ypu> |
| Component: | kernel | Assignee: | Amit Shah <amit.shah> |
| Status: | CLOSED WORKSFORME | QA Contact: | Virtualization Bugs <virt-bugs> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 6.2 | CC: | areis, juzhang, mkenneth, shuang, tburke, virt-maint |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2012-01-12 08:33:11 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Joy Pu
2011-08-15 04:57:31 UTC
These are warnings where it looks like creating sysfs files takes some time, while the port is already yanked off. When you say 'guest cannot init virtio serial port', what do you mean exactly? Do you not see entries in /dev/vport*? Something else? The fix will be to take a lock in the port init routine and ensure the init has finished before we delete the port or add a new one at the same id. (In reply to comment #2) > These are warnings where it looks like creating sysfs files takes some time, > while the port is already yanked off. > > When you say 'guest cannot init virtio serial port', what do you mean exactly? > Do you not see entries in /dev/vport*? Something else? > This is hard to test. When I add "ls -l /dev/vport*" to check it after hotplug its actions changes like with sleep 1s. I think the problem is just as you said. hotplug a device will return before everything in guest is OK. So hotplug and un-hotplug without sleep will interrupt the init process in guest system. > The fix will be to take a lock in the port init routine and ensure the init has > finished before we delete the port or add a new one at the same id. This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux maintenance release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux Update release for currently deployed products. This request is not yet committed for inclusion in an Update release. > In this version, still can see the warning message from guest serial port log,
> but the scripts will report init and delete virtio serial console succeed.
Can you give more details?
Which warning message do you see?
Also, the script reports init and delete succeeded after 100 non-stop runs? What used to happen in the original kernel? Did the script just stop? Guest freeze?
(In reply to comment #8) > > In this version, still can see the warning message from guest serial port log, > > but the scripts will report init and delete virtio serial console succeed. > > Can you give more details? > > Which warning message do you see? > > Also, the script reports init and delete succeeded after 100 non-stop runs? > What used to happen in the original kernel? Did the script just stop? Guest > freeze? Sorry, forget to upgrade guest's kernel. After guest kernel update, the warning message is gone. In the scripts it only judge the init and delete with monitor command output. So in bug report version of qemu kvm and kernel. guest will show warning message and freeze for a while which will lead the monitor command failed and warning message in guest. After update host kernel and qemu kvm, the guest kernel only show warning message but alive. So only warning message shows but monitor command return no error. After update guest kernel the warning message didn't show up and the monitor command return no error. This is the test version: host & guest kernel version: 2.6.32-223.el6.test.x86_64 qemu and kvm version: rpm -qa|grep qemu qemu-img-0.12.1.2-2.213.el6.x86_64 qemu-kvm-0.12.1.2-2.213.el6.x86_64 qemu-kvm-debuginfo-0.12.1.2-2.213.el6.x86_64 qemu-kvm-tools-0.12.1.2-2.213.el6.x86_64 gpxe-roms-qemu-0.9.7-6.7.el6.noarch (In reply to comment #9) > Sorry, forget to upgrade guest's kernel. After guest kernel update, the warning > message is gone. Ah OK, that's much better. > In the scripts it only judge the init and delete with monitor command output. > So in bug report version of qemu kvm and kernel. guest will show warning > message and freeze for a while which will lead the monitor command failed and > warning message in guest. After update host kernel and qemu kvm, the guest > kernel only show warning message but alive. So only warning message shows but > monitor command return no error. After update guest kernel the warning message > didn't show up and the monitor command return no error. For this test, only guest kernel update is required; host qemu and host kernel don't matter. So can you run a last quick test: just test with buggy as well as new build of guest kernel without modifying host packages and let me know if with the new kernel warnings as well as freezes are gone? Thanks for the testing! (In reply to comment #10) > (In reply to comment #9) > > Sorry, forget to upgrade guest's kernel. After guest kernel update, the warning > > message is gone. > > Ah OK, that's much better. > > > In the scripts it only judge the init and delete with monitor command output. > > So in bug report version of qemu kvm and kernel. guest will show warning > > message and freeze for a while which will lead the monitor command failed and > > warning message in guest. After update host kernel and qemu kvm, the guest > > kernel only show warning message but alive. So only warning message shows but > > monitor command return no error. After update guest kernel the warning message > > didn't show up and the monitor command return no error. > > For this test, only guest kernel update is required; host qemu and host kernel > don't matter. > > So can you run a last quick test: just test with buggy as well as new build of > guest kernel without modifying host packages and let me know if with the new > kernel warnings as well as freezes are gone? > > Thanks for the testing! Welcome and had more tests with this case with new guest kernel and both old host kernel and new host kernel. After update guest kernel, seems guest will not freeze again. But the warning message still can be triggered in both old host kernel and new host kernel.The difference is old host kernel seems more easily to trigger the init failed warning in guest. With new host kernel and the warning message is still shows up sometime. 1000 times hotplug it shows 7 times. In the old host kernel. The warning message is easily shows up. For 100 times hotplug, it will show up 8~9 times in average. Warning message is : 2012-01-06 19:28:30: ------------[ cut here ]------------ 2012-01-06 19:28:30: WARNING: at fs/sysfs/dir.c:512 sysfs_add_one+0xc9/0x130() (Tainted: G W ---------------- ) 2012-01-06 19:28:30: Hardware name: KVM 2012-01-06 19:28:30: sysfs: cannot create duplicate filename '/devices/pci0000:00/0000:00:05.0/virtio2/virtio-ports/vport0p2/name' 2012-01-06 19:28:30: Modules linked in: sunrpc ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log uinput ppdev parport_pc parport sg microcode virtio_console virtio_net i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded: mperf] 2012-01-06 19:28:30: Pid: 7, comm: events/0 Tainted: G W ---------------- 2.6.32-223.el6.test.x86_64 #1 2012-01-06 19:28:30: Call Trace: 2012-01-06 19:28:30: [<ffffffff81069997>] ? warn_slowpath_common+0x87/0xc0 2012-01-06 19:28:30: [<ffffffff81069a86>] ? warn_slowpath_fmt+0x46/0x50 2012-01-06 19:28:30: [<ffffffff811ed599>] ? sysfs_add_one+0xc9/0x130 2012-01-06 19:28:30: [<ffffffff811eb882>] ? sysfs_add_file_mode+0x62/0xb0 2012-01-06 19:28:30: [<ffffffff811eeb71>] ? internal_create_group+0xc1/0x1a0 2012-01-06 19:28:30: [<ffffffff811eec83>] ? sysfs_create_group+0x13/0x20 2012-01-06 19:28:30: [<ffffffffa0134fc0>] ? control_work_handler+0x1a0/0x498 [virtio_console] 2012-01-06 19:28:30: [<ffffffffa0134e20>] ? control_work_handler+0x0/0x498 [virtio_console] 2012-01-06 19:28:30: [<ffffffff8108b0d0>] ? worker_thread+0x170/0x2a0 2012-01-06 19:28:30: [<ffffffff81090a10>] ? autoremove_wake_function+0x0/0x40 2012-01-06 19:28:30: [<ffffffff8108af60>] ? worker_thread+0x0/0x2a0 2012-01-06 19:28:30: [<ffffffff810906a6>] ? kthread+0x96/0xa0 2012-01-06 19:28:30: [<ffffffff8100c14a>] ? child_rip+0xa/0x20 2012-01-06 19:28:30: [<ffffffff81090610>] ? kthread+0x0/0xa0 2012-01-06 19:28:30: [<ffffffff8100c140>] ? child_rip+0x0/0x20 2012-01-06 19:28:30: ---[ end trace 93ec87a4da16c595 ]--- 2012-01-06 19:28:30: virtio-ports vport0p2: Error -17 creating sysfs device attributes OK, as long as the guest doesn't freeze, we should be fine. As for triggering this sysfs warning, I'll have to see if there's a way to wait on the sysfs path really going away at unplug time. Joy has been testing this bug in various scenarios (smp 1, smp 3) and looks like it's no longer reproducible with RHEL6.1 nor Fedora 16 guests. We don't know yet why it's no longer reproducible, but the only change was the host s/w was upgraded to RHEL6.2. We can try reproducing on a RHEL6.1 host, but that'll just be an academic exercise. I'll close the bug for now; if this shows up again, we can work on it. |