Bug 1069351
| Summary: | Endless stream of error dialogs when libvirtd stops/crashes on remote host | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Dr. David Alan Gilbert <dgilbert> | ||||
| Component: | virt-manager | Assignee: | Cole Robinson <crobinso> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||
| Severity: | unspecified | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 20 | CC: | berrange, crobinso, info, mkelly, rhack, vg.aetera, virt-maint, vyasevic | ||||
| Target Milestone: | --- | Keywords: | Reopened | ||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | virt-manager-1.0.1-2.fc20 | Doc Type: | Bug Fix | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2014-04-25 04:29:47 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
|
Description
Dr. David Alan Gilbert
2014-02-24 19:46:48 UTC
Thanks for the report. That error usually means that libvirtd crashed, we should be closing the connection here but the recent release regressed. Fixed upstream now:
commit ce64d037bff56db994fedd065a9a34b8e827dda2
Author: Cole Robinson <crobinso>
Date: Tue Feb 25 15:17:34 2014 -0500
engine: Fix closing connection when tick() fails (bz 1069351)
Thanks. I don't think the remote libvirtd actually died, I suspect this was just the remote being slow to respond while stuffing the initramfs down to it over a congested connection. Fixed in virt-manager-1.0.0-3.fc21 virt-manager-1.0.0-3.fc20 has been submitted as an update for Fedora 20. https://admin.fedoraproject.org/updates/virt-manager-1.0.0-3.fc20 Ignore the 'rawhide' comment Package virt-manager-1.0.0-3.fc20: * should fix your issue, * was pushed to the Fedora 20 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing virt-manager-1.0.0-3.fc20' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2014-3272/virt-manager-1.0.0-3.fc20 then log in and leave karma (feedback). virt-manager-1.0.0-3.fc20 has been pushed to the Fedora 20 stable repository. If problems still persist, please make note of it in this bug report. Just hit it on -3; I was doing a yum update on one of the remote hosts, so I guess
that restarted libvirtd.
[Mon, 10 Mar 2014 11:58:32 virt-manager 3099] DEBUG (console:271) Creating SSH tunnel: ssh -l root vl403 sh -c 'nc -q 2>&1 | grep "requires an argument" >/dev/null;if [ $? -eq 0 ] ; then CMD="nc -q 0 127.0.0.1 5900";else CMD="nc 127.0.0.1 5900";fi;eval "$CMD";'
[Mon, 10 Mar 2014 11:58:32 virt-manager 3099] DEBUG (console:291) Open tunnel PID=11763 OUTFD=89 ERRFD=91
[Mon, 10 Mar 2014 12:00:21 virt-manager 3099] ERROR (engine:371) Could not refresh connection qemu+ssh://root@vl403/system
Traceback (most recent call last):
File "/usr/share/virt-manager/virtManager/engine.py", line 349, in _tick_single_conn
conn.tick(**kwargs)
File "/usr/share/virt-manager/virtManager/connection.py", line 1150, in tick
self.hostinfo = self._backend.getInfo()
File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3652, in getInfo
if ret is None: raise libvirtError ('virNodeGetInfo() failed', conn=self)
libvirtError: End of file while reading data: Ncat: Broken pipe.: Input/output error
[Mon, 10 Mar 2014 12:00:21 virt-manager 3099] DEBUG (engine:372) Closing connection since libvirtd appears to have stopped
[Mon, 10 Mar 2014 12:00:24 virt-manager 3099] DEBUG (error:84) error dialog message:
summary=Error polling connection 'qemu+ssh://root@vl403/system': internal error: client socket is closed
details=Error polling connection 'qemu+ssh://root@vl403/system': internal error: client socket is closed
[Mon, 10 Mar 2014 12:00:27 virt-manager 3099] DEBUG (error:84) error dialog message:
summary=Error polling connection 'qemu+ssh://root@vl403/system': internal error: client socket is closed
details=Error polling connection 'qemu+ssh://root@vl403/system': internal error: client socket is closed
and then it was spitting another dialog about once every 3 seconds.
Yeah, sorry, I messed up the first patch. The test scenario I used was pretty contrived, I should have just done the expected test of shutting down libvirtd while virt-manager is connected.
commit 081e34715ffa5a210e1e0c8670fe3a1a3ec5180b
Author: Cole Robinson <crobinso>
Date: Mon Mar 10 09:33:04 2014 -0400
connection: Handle errors when deregistering events on close (bz 1069351)
Otherwise this interrupts the close/cleanup routine, and the connection
never appears to disconnect in the UI. This causes error dialog spamming
when libvirtd goes down.
*** Bug 1074856 has been marked as a duplicate of this bug. *** Package virt-manager-1.0.0-6.fc20: * should fix your issue, * was pushed to the Fedora 20 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing virt-manager-1.0.0-6.fc20' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2014-3775/virt-manager-1.0.0-6.fc20 then log in and leave karma (feedback). *** Bug 1075815 has been marked as a duplicate of this bug. *** *** Bug 1076155 has been marked as a duplicate of this bug. *** Tried the package from testing. Seems to work for me. Thanks virt-manager-1.0.0-6.fc20 from comment #11 fixed the problem for me, too. virt-manager-1.0.0-6.fc20 has been pushed to the Fedora 20 stable repository. If problems still persist, please make note of it in this bug report. I'm running this version of virt-manager but still have many dialogs spawning when the connection closes: When my VPN connection closes while having a remote console open in virt-manager, I get 10-20 or sometimes even more dialog pop-ups with the same error message "Error polling connection 'qemu+ssh://xxx@yyyy/system': 'NoneType' object has no attribute 'getinfo'" It's annoying but a killall -9 virt-manager saves me from closing all the dialog windows. Version-Release number of selected component (if applicable): virt-manager-1.0.0-6.fc20.noarc How reproducible: Everytime Steps to Reproduce: 1. Connect to remote location with vpn 2. open virt-manager and connect to remote server 3. open console of a VM 4. disconnect VPN There is a newer version of virt-manager in updates, 1.0.1, that had additional fixes. Please update to that version, fully restart virt-manager, and see if the issue persists. Issue persists with virt-manager-1.0.1-1.fc20.noarch. The symptom here is a little different. The connection is reset and when virt-manager detects this, we get many spawned dialogs of 2 different kinds: First: Error polling connection 'qemu:///system': internal error: client socket is closed Second: Error polling connection 'qemu+ssh://root.eng.bos.redhat.com/system': 'NoneType' object has no attribute 'getInfo' Sometimes the second dialog doesn't show up. -vlad FWIW I can't reproduce the issue with latest virt-manager. Please ensure virt-manager has been fully restarted since updating. If the issue still persists, please provide virt-manager --debug output when reproducing Created attachment 886707 [details]
virt-manager --debug log
I managed to isolate the reproduction scenario:
1) Connect virt-manager to the remote system via ssh.
2) Suspect the system running virt-manager
3) Shutdown the remote system.
4) Wake up system running virt-manager
- at this point virt-manager things it's still connected.
5) Boot up the remote system again
- When the system comes up the ssh socket is reset
6) Close a bunch of dialogs.
I've attached the output of virt-manager --debug.
-vlad
thanks vlad, I managed to reproduce and fix upstream. Reopening this virt-manager-1.0.1-2.fc20 has been submitted as an update for Fedora 20. https://admin.fedoraproject.org/updates/virt-manager-1.0.1-2.fc20 Package virt-manager-1.0.1-2.fc20: * should fix your issue, * was pushed to the Fedora 20 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing virt-manager-1.0.1-2.fc20' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2014-5335/virt-manager-1.0.1-2.fc20 then log in and leave karma (feedback). virt-manager-1.0.1-2.fc20 has been pushed to the Fedora 20 stable repository. If problems still persist, please make note of it in this bug report. |