Bug 1200991
| Summary: | Create directory for lease files if it's missing (i.e. if no net is autostarted) | |||
|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 6 | Reporter: | Michael Everette <meverett> | |
| Component: | libvirt | Assignee: | Michal Privoznik <mprivozn> | |
| Status: | CLOSED ERRATA | QA Contact: | Pavel Stehlik <pstehlik> | |
| Severity: | high | Docs Contact: | ||
| Priority: | high | |||
| Version: | 6.7 | CC: | bazulay, danken, dkuznets, dyuan, ecohen, gklein, iheim, jherrman, lpeer, lsurette, meverett, mprivozn, mzhan, oourfali, rbalakri, tlavigne, ybronhei, yeylon | |
| Target Milestone: | rc | Keywords: | ZStream | |
| Target Release: | 6.7 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | infra | |||
| Fixed In Version: | libvirt-0.10.2-52.el6 | Doc Type: | Bug Fix | |
| Doc Text: |
Prior to this update, when starting the libvirt network filter (nwfilter) while no network was available, the nwfilter driver initialization failed. With this update, nwfilter automatically creates a path for the nwfilter.leases file. This allows nwfilter to be independent from the network driver and to be able to start with no network available.
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 1208619 (view as bug list) | Environment: | ||
| Last Closed: | 2015-07-22 05:49:01 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1208619 | |||
*** Bug 1200994 has been marked as a duplicate of this bug. *** please attach libvirt.log and vdsm.log to the bugzilla Anything else needed Yaniv? you keep having this error in libvirtd.log: 2015-03-07 13:12:35.461+0000: 3818: info : libvirt version: 0.10.2, package: 46.el6_6.3 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2015-01-23-10:39:40, x86-026.build.eng.bos.redhat.com) 2015-03-07 13:12:35.461+0000: 3818: error : virNWFilterSnoopLeaseFileRefresh:1903 : open("/var/run/libvirt/network/nwfilter.ltmp"): No such file or directory 2015-03-07 13:12:37.126+0000: 3818: error : qemudStartup:823 : unable to set ownership of '/var/cache/libvirt/qemu' to 107:107: No such file or directory 2015-03-07 13:12:37.126+0000: 3818: error : virStateInitialize:798 : Initialization of QEMU state driver failed 2015-03-07 13:12:37.126+0000: 3818: error : daemonRunStateInit:740 : Driver state initialization failed And libvirtd doesn't start properly. It has nothing to do with vdsm please read https://libvirt.org/drvqemu.html and validate that qemu drivers are set properly. Danken, It looks like libvirt service was down ? Anyway according to comment #6 it looks like it may be related to the NWFilters, Can you please take a look? We define a nwfilter on vdsmd startup, but I doubt if it has anything to do with libvirtd failing to start. Michael, would you make sure that libvirtd is stopped, and then try running it from the command line (as root) libvirtd --listen & does it start properly? Does it respond to `virsh -r capabilities`? mprivozn, do you have an idea what makes libvirtd fail to start here? (In reply to Dan Kenigsberg from comment #8) > > mprivozn, do you have an idea what makes libvirtd fail to start here? I do actually. It's because a long ago (in 2013) we had a bug in NWFilter implementation. This commit needs to be backported into RHEL6: http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=dc4cdc57;hp=cbe31911ad893579cbeac6d646c3a2649bf21454 Should I switch this bug over to libvirt or clone it? Dan, Do you need me to still have customer run those or is what Micahl pointing to the issue here and we need to move this bz? Michael, it seems Michal recognizes the bug and does not need further info. Michal, I don't think this bug needs tracking on RHEV-M, so I'm moving it to libvirt. Can you tell how come we don't always see this bug? (In reply to Dan Kenigsberg from comment #11) > Michael, it seems Michal recognizes the bug and does not need further info. > > Michal, I don't think this bug needs tracking on RHEV-M, so I'm moving it to > libvirt. > > Can you tell how come we don't always see this bug? If there's an autostarted network in the libvirt configs, the path is created. And since networks are started prior to NWFilters, the path was there. However, if there's no network to autostart, the path wasn't created nad NWFilter failed. According to comment 12, I can reproduce it. [root@localhost ~]# rpm -q libvirt libvirt-0.10.2-51.el6.x86_64 Before restarting host OS, disabled all autostart virtual networks and removed /var/run/libvirt/network folder [root@localhost ~]# virsh net-list --all Name State Autostart Persistent -------------------------------------------------- default active no yes [root@localhost ~]# ll /var/run/libvirt total 4 srwxrwxrwx. 1 root root 0 Apr 2 18:10 libvirt-sock srwxrwxrwx. 1 root root 0 Apr 2 18:10 libvirt-sock-ro drwxr-xr-x. 2 root root 4096 Apr 2 18:02 qemu After starting the host OS again: [root@localhost ~]# cat /var/log/libvirt/libvirtd.log | grep "virNWFilterSnoopLeaseFileRefresh" -b5 1072751-2015-04-02 10:10:10.216+0000: 2637: debug : udevProcessDeviceListEntry:1387 : Failed to create node device for udev device '/sys/devices/pci0000:00/0000:00:1b.0/sound/card0/controlC0' 1072935-2015-04-02 10:10:10.216+0000: 2637: debug : virStateInitialize:795 : Running global init for Secret state driver 1073048-2015-04-02 10:10:10.216+0000: 2637: debug : virStateInitialize:795 : Running global init for NWFilter state driver 1073163-2015-04-02 10:10:10.216+0000: 2637: debug : virNWFilterLearnInit:791 : Initializing IP address learning 1073267-2015-04-02 10:10:10.216+0000: 2637: debug : virNWFilterDHCPSnoopInit:2080 : Initializing DHCP snooping 1073370:2015-04-02 10:10:10.216+0000: 2637: error : virNWFilterSnoopLeaseFileRefresh:1903 : open("/var/run/libvirt/network/nwfilter.ltmp"): No such file or directory 1073528-2015-04-02 10:10:10.216+0000: 2637: debug : virNWFilterTechDriversInit:60 : Initializing NWFilter technology drivers 1073645-2015-04-02 10:10:10.217+0000: 2637: debug : virCommandRunAsync:2229 : About to run /bin/sh -c 'EBT="/sbin/ebtables" 1073761-cmd='\''$EBT -t nat -L'\'' 1073788-eval res=\$\("${cmd} 2>&1"\) 1073817-if [ $? -ne 0 ]; then echo "Failure to execute command '\''${cmd}'\'' : '\''${res}'\''."; exit 1;fi [root@localhost ~]# ll /var/run/libvirt total 4 srwxrwxrwx. 1 root root 0 Apr 2 18:10 libvirt-sock srwxrwxrwx. 1 root root 0 Apr 2 18:10 libvirt-sock-ro drwxr-xr-x. 2 root root 4096 Apr 2 18:02 qemu [root@localhost ~]# virsh dumpxml aa | grep "/interface" -B8 <interface type='network'> <mac address='52:54:00:9c:0e:95'/> <source network='default'/> <model type='virtio'/> <filterref filter='clean-traffic'> <parameter name='CTRL_IP_LEARNING' value='dhcp'/> </filterref> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> [root@localhost ~]# virsh net-start default Network default started [root@localhost ~]# ll /var/run/libvirt/network/ total 4 -rw-r--r--. 1 root root 5 Apr 2 18:11 default.pid [root@localhost ~]# virsh start aa Domain aa started [root@localhost ~]# ebtables -t nat -L Bridge table: nat Bridge chain: PREROUTING, entries: 1, policy: ACCEPT -i vnet0 -j libvirt-I-vnet0 Bridge chain: OUTPUT, entries: 0, policy: ACCEPT Bridge chain: POSTROUTING, entries: 1, policy: ACCEPT -o vnet0 -j libvirt-O-vnet0 Bridge chain: libvirt-I-vnet0, entries: 2, policy: ACCEPT -p IPv4 -s 52:54:0:9c:e:95 --ip-proto udp --ip-sport 68 --ip-dport 67 -j ACCEPT -j DROP Bridge chain: libvirt-O-vnet0, entries: 3, policy: ACCEPT -p IPv4 -d 52:54:0:9c:e:95 --ip-proto udp --ip-sport 67 --ip-dport 68 -j ACCEPT -p IPv4 -d Broadcast --ip-proto udp --ip-sport 67 --ip-dport 68 -j ACCEPT -j DROP In guest OS: [root@localhost ~]# virsh console aa Connected to domain aa Escape character is ^] Red Hat Enterprise Linux Server release 6.6 (Santiago) Kernel 2.6.32-504.el6.x86_64 on an x86_64 sowang login: root Password: Last login: Thu Apr 2 16:43:49 on tty1 [root@sowang ~]# ip ad 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 52:54:00:9c:0e:95 brd ff:ff:ff:ff:ff:ff inet 192.168.122.235/24 brd 192.168.122.255 scope global eth0 inet6 fe80::5054:ff:fe9c:e95/64 scope link valid_lft forever preferred_lft forever Verified as below:
Before rebooting OS, removed "network" folder under /var/run/libvirt
[root@intel-e31225-8-3 ~]# rpm -q libvirt
libvirt-0.10.2-53.el6.x86_64
[root@intel-e31225-8-3 ~]# virsh net-list --all
Name State Autostart Persistent
--------------------------------------------------
default active no yes
[root@intel-e31225-8-3 ~]# ll /var/run/libvirt
total 4
srwxrwxrwx. 1 root root 0 Apr 13 10:27 libvirt-sock
srwxrwxrwx. 1 root root 0 Apr 13 10:27 libvirt-sock-ro
drwxr-xr-x. 2 root root 4096 Apr 13 10:27 qemu
[root@intel-e31225-8-3 ~]# reboot
Broadcast message from root.nay.redhat.com
(/dev/pts/0) at 10:29 ...
The system is going down for reboot NOW!
[root@intel-e31225-8-3 ~]# Connection to 10.66.85.74 closed by remote host.
Connection to 10.66.85.74 closed.
After booting up OS:
[hujianwei@localhost libvirt]$ ssh root.85.74 -X
root.85.74's password:
Last login: Fri Apr 10 15:06:42 2015 from 10.66.7.130
[root@intel-e31225-8-3 ~]# cat /var/log/libvirt/libvirtd.log | grep "virNWFilterSnoopLeaseFileRefresh" -b5
<=== No error output
[root@intel-e31225-8-3 ~]# ll /var/run/libvirt
total 8
srwxrwxrwx. 1 root root 0 Apr 13 10:32 libvirt-sock
srwxrwxrwx. 1 root root 0 Apr 13 10:32 libvirt-sock-ro
drwx------. 2 root root 4096 Apr 13 10:32 network
drwxr-xr-x. 2 root root 4096 Apr 13 10:30 qemu
[root@intel-e31225-8-3 ~]# ll /var/run/libvirt/network/
total 0
-rw-r--r--. 1 root root 0 Apr 13 10:32 nwfilter.leases
[root@intel-e31225-8-3 ~]# virsh net-list --all
Name State Autostart Persistent
--------------------------------------------------
default inactive no yes
[root@intel-e31225-8-3 ~]# virsh list --all
Id Name State
----------------------------------------------------
- virt-tests-vm1 shut off
[root@intel-e31225-8-3 ~]# virsh dumpxml virt-tests-vm1 | grep "/interface" -B8
<interface type='network'>
<mac address='52:54:00:88:89:8a'/>
<source network='default'/>
<model type='virtio'/>
<filterref filter='clean-traffic'>
<parameter name='CTRL_IP_LEARNING' value='dhcp'/>
</filterref>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
[root@intel-e31225-8-3 ~]# virsh net-start default
Network default started
[root@intel-e31225-8-3 ~]# virsh start virt-tests-vm1
Domain virt-tests-vm1 started
[root@intel-e31225-8-3 ~]# cat /var/run/libvirt/network/nwfilter.leases
1428896639 a62c462d-bd34-742d-943a-643ba6322da7-52:54:00:88:89:8A 192.168.122.166 192.168.122.1
[root@intel-e31225-8-3 ~]# ebtables -t nat -L
Bridge table: nat
Bridge chain: PREROUTING, entries: 1, policy: ACCEPT
-i vnet0 -j libvirt-I-vnet0
Bridge chain: OUTPUT, entries: 0, policy: ACCEPT
Bridge chain: POSTROUTING, entries: 1, policy: ACCEPT
-o vnet0 -j libvirt-O-vnet0
Bridge chain: libvirt-I-vnet0, entries: 9, policy: ACCEPT
-j I-vnet0-mac
-p IPv4 -j I-vnet0-ipv4-ip
-p IPv4 -j ACCEPT
-p ARP -j I-vnet0-arp-mac
-p ARP -j I-vnet0-arp-ip
-p ARP -j ACCEPT
-p 0x8035 -j I-vnet0-rarp
-p 0x835 -j ACCEPT
-j DROP
Bridge chain: libvirt-O-vnet0, entries: 4, policy: ACCEPT
-p IPv4 -j O-vnet0-ipv4
-p ARP -j ACCEPT
-p 0x8035 -j O-vnet0-rarp
-j DROP
Bridge chain: I-vnet0-mac, entries: 2, policy: ACCEPT
-s 52:54:0:88:89:8a -j RETURN
-j DROP
Bridge chain: I-vnet0-ipv4-ip, entries: 3, policy: ACCEPT
-p IPv4 --ip-src 0.0.0.0 --ip-proto udp -j RETURN
-p IPv4 --ip-src 192.168.122.166 -j RETURN
-j DROP
Bridge chain: O-vnet0-ipv4, entries: 1, policy: ACCEPT
-j ACCEPT
Bridge chain: I-vnet0-arp-mac, entries: 2, policy: ACCEPT
-p ARP --arp-mac-src 52:54:0:88:89:8a -j RETURN
-j DROP
Bridge chain: I-vnet0-arp-ip, entries: 2, policy: ACCEPT
-p ARP --arp-ip-src 192.168.122.166 -j RETURN
-j DROP
Bridge chain: I-vnet0-rarp, entries: 2, policy: ACCEPT
-p 0x8035 -s 52:54:0:88:89:8a -d Broadcast --arp-op Request_Reverse --arp-ip-src 0.0.0.0 --arp-ip-dst 0.0.0.0 --arp-mac-src 52:54:0:88:89:8a --arp-mac-dst 52:54:0:88:89:8a -j ACCEPT
-j DROP
Bridge chain: O-vnet0-rarp, entries: 2, policy: ACCEPT
-p 0x8035 -d Broadcast --arp-op Request_Reverse --arp-ip-src 0.0.0.0 --arp-ip-dst 0.0.0.0 --arp-mac-src 52:54:0:88:89:8a --arp-mac-dst 52:54:0:88:89:8a -j ACCEPT
-j DROP
We can get expected results, moved to Verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2015-1252.html |
Description of problem: vdsmd service is not starting up automatically on system reboot. Customer has to manually restart the service each time when system is rebooted. Version-Release number of selected component (if applicable): RHEV-M Version : 3.5.0-0.32.el6ev Host: RHEL-6Server-6.6.0.2.el6 vdsm-4.16.8.1-8.el6ev.x86_64 kernel: 2.6.32-504.8.1.el6.x86_64 How reproducible: very Steps to Reproduce: 1. reboot 2. vdsm fails to start 3. message seen: vdsm: Running mkdirs vdsm: Running configure_coredump vdsm: Running configure_vdsm_logs vdsm: Running wait_for_network vdsm: Running run_init_hooks vdsm: Running upgraded_version_check vdsm: Running check_is_configured libvirt is already configured for vdsm vdsm: Running validate_configuration SUCCESS: ssl configured to true. No conflicts vdsm: Running prepare_transient_repository vdsm: Running syslog_available vdsm: Running nwfilter libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory Traceback (most recent call last): File "/usr/bin/vdsm-tool", line 209, in main return tool_command[cmd]["command"](*args) File "/usr/lib/python2.6/site-packages/vdsm/tool/nwfilter.py", line 39, in main conn = libvirtconnection.get(None, False) File "/usr/lib/python2.6/site-packages/vdsm/libvirtconnection.py", line 151, in get conn = _open_qemu_connection() File "/usr/lib/python2.6/site-packages/vdsm/libvirtconnection.py", line 95, in _open_qemu_connection return utils.retry(libvirtOpen, timeout=10, sleep=0.2) File "/usr/lib/python2.6/site-packages/vdsm/utils.py", line 1087, in retry return func() File "/usr/lib64/python2.6/site-packages/libvirt.py", line 102, in openAuth if ret is None:raise libvirtError('virConnectOpenAuth() failed') libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory vdsm: stopped during execute nwfilter task (task returned with Issuing a manual restart and vdsm comes up fine. Could this be a regression to https://bugzilla.redhat.com/show_bug.cgi?id=866538 ?