Created attachment 1317495 [details] supervdsm log Description of problem: LLDP Protocol - EnableLldpError: Failed to enable LLDP on <Port> in supervdsm logs - supervdsm.log is spammed with lldp timeout errors : ainThread::ERROR::2017-08-24 10:41:04,484::initializer::53::root::(_lldp_init) Failed to enable LLDP on ens2f1 Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/network/initializer.py", line 51, in _lldp_init Lldp.enable_lldp_on_iface(device) File "/usr/lib/python2.7/site-packages/vdsm/network/lldp/lldpad.py", line 30, in enable_lldp_on_iface lldptool.enable_lldp_on_iface(iface, rx_only) File "/usr/lib/python2.7/site-packages/vdsm/network/lldpad/lldptool.py", line 46, in enable_lldp_on_iface raise EnableLldpError(rc, out, err, iface) EnableLldpError: (1, "timeout\n'M00000001C3040000000c06ens2f1000badminStatus0002rx' command timed out.\n", '', 'ens2f1') systemctl status lldpad ● lldpad.service - Link Layer Discovery Protocol Agent Daemon. Loaded: loaded (/usr/lib/systemd/system/lldpad.service; disabled; vendor preset: disabled) Active: active (running) since Thu 2017-08-17 17:25:15 IDT; 6 days ago Main PID: 1869 (lldpad) CGroup: /system.slice/lldpad.service └─1869 /usr/sbin/lldpad -t [root@navy-vds1 ~]# lldptool set-lld -i ens2f1 adminStatus=rxtx adminStatus = rxtx [root@navy-vds1 ~]# lldptool get-lld -i ens2f1 adminStatus adminStatus=rxtx [root@navy-vds1 ~]# lldptool get-tlv -n -i ens2f1 Chassis ID TLV MAC: 18:ef:63:a1:75:0c Port ID TLV Local: Gi0/12 Time to Live TLV 120 System Name TLV rack03-sw02-lab4.tlv.redhat.com System Description TLV Cisco IOS Software, C3560 Software (C3560-ADVIPSERVICESK9-M), Version 12.2(44)SE6, RELEASE SOFTWARE (fc1) Copyright (c) 1986-2009 by Cisco Systems, Inc. Compiled Mon 09-Mar-09 17:42 by gereddy Port Description TLV GigabitEthernet0/12 System Capabilities TLV System capabilities: Bridge, Router Enabled capabilities: Bridge Port VLAN ID TLV PVID: 161 MAC/PHY Configuration Status TLV Auto-negotiation supported and enabled PMD auto-negotiation capabilities: 0xc036 MAU type: 1000 BaseTFD End of LLDPDU TLV Version-Release number of selected component (if applicable): vdsm-4.20.2-111.gita9f6d88.el7.centos.x86_64 How reproducible: Seems to be always in the background on all hosts, although the lldpad service is running and the port status is enabled.
I think we should require selinux-policy-targeted >= 3.13.1-166.el7_4.3 explicitly. Michael, can you check if it solves this?
(In reply to Dan Kenigsberg from comment #1) > I think we should require selinux-policy-targeted >= 3.13.1-166.el7_4.3 > explicitly. > > Michael, can you check if it solves this? Dan we already running selinux-policy-targeted-3.13.1-166.el7.noarch
The trailing _4.3 in the rpm is the important one, in this context. It tells that I'm asking you to take the rhel-7.4.1 package, not the plain 7.4.0 one.
(In reply to Dan Kenigsberg from comment #3) > The trailing _4.3 in the rpm is the important one, in this context. It tells > that I'm asking you to take the rhel-7.4.1 package, not the plain 7.4.0 one. I can't find such package any where. Please provide me one if you want me to test this. I can only find our latest which we already running.
Hi Dominik, As we saw together on my hosts, the next messages appears right after we installing selinux-policy-targeted-3.13.1-166.el7_4.3.noarch on all of the ports- MainThread::ERROR::2017-08-28 07:44:09,734::initializer::53::root::(_lldp_init) Failed to enable LLDP on enp4s0 Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/network/initializer.py", line 51, in _lldp_init Lldp.enable_lldp_on_iface(device) File "/usr/lib/python2.7/site-packages/vdsm/network/lldp/lldpad.py", line 30, in enable_lldp_on_iface lldptool.enable_lldp_on_iface(iface, rx_only) File "/usr/lib/python2.7/site-packages/vdsm/network/lldpad/lldptool.py", line 46, in enable_lldp_on_iface raise EnableLldpError(rc, out, err, iface) EnableLldpError: (255, '', 'connect: Connection refused\nFailed to connect to lldpad - clif_open: Invalid argument\n', 'enp4s0') Do we ok with it? If yes, then those messages should be removed from the log and handled by vdsm.
I expect that this is related to update process of SELinux rules. Are the messages produced after reboot?
(In reply to Dominik Holler from comment #9) > I expect that this is related to update process of SELinux rules. > Are the messages produced after reboot? Yes the messages are still produced indeed, i have updated to selinux-policy-targeted-3.13.1-166.el7_4.3.noarch few days ago(also rebooted the host) and i still see this error messages in supervdsm log(for example from last night) - MainThread::ERROR::2017-08-30 22:26:24,711::initializer::53::root::(_lldp_init) Failed to enable LLDP on enp6s0f0 Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/network/initializer.py", line 51, in _lldp_init Lldp.enable_lldp_on_iface(device) File "/usr/lib/python2.7/site-packages/vdsm/network/lldp/lldpad.py", line 30, in enable_lldp_on_iface lldptool.enable_lldp_on_iface(iface, rx_only) File "/usr/lib/python2.7/site-packages/vdsm/network/lldpad/lldptool.py", line 46, in enable_lldp_on_iface raise EnableLldpError(rc, out, err, iface) EnableLldpError: (255, '', 'connect: Connection refused\nFailed to connect to lldpad - clif_open: Invalid argument\n', 'enp6s0f0') rpm -qa | grep target selinux-policy-targeted-3.13.1-166.el7_4.3.noarch Also note, that this error messages appear both on hosts with the updated package and without.
Also with the newer version selinux-policy-targeted-3.13.1-166.el7_4.4.noarch
> 'connect: Connection refused\nFailed to connect to lldpad - clif_open: Invalid argument\n', 'enp6s0f0' This is a other error message, maybe we should track this in a new bug.
(In reply to Dominik Holler from comment #12) > > 'connect: Connection refused\nFailed to connect to lldpad - clif_open: Invalid argument\n', 'enp6s0f0' > > This is a other error message, maybe we should track this in a new bug. This are the error messages we see after package update. We don't see the original error message(about the timeout) in comment#0 after updating the package , but we do see this new error messages. If you think we need a new bug i don't mind, let me know what to do.
(In reply to Michael Burman from comment #13) > (In reply to Dominik Holler from comment #12) > > > 'connect: Connection refused\nFailed to connect to lldpad - clif_open: Invalid argument\n', 'enp6s0f0' > > > > This is a other error message, maybe we should track this in a new bug. > > This are the error messages we see after package update. We don't see the > original error message(about the timeout) in comment#0 after updating the > package , but we do see this new error messages. > If you think we need a new bug i don't mind, let me know what to do. New bug has been reported for the new errors after selinux-policy-targeted update , see BZ 1487078
Thanks for filing a fresh bug. Please reopen this bug if you see other "timeout" error on a host that was booted with el7.4.1's selinux-policy.
Verified with - vdsm-4.19.30-1.el7ev.x86_64 selinux-policy-targeted-3.13.1-166.el7_4.4.noarch selinux-policy-3.13.1-166.el7_4.4.noarch