Bug 2084171 - [c9s] failing to setup network
Summary: [c9s] failing to setup network
Keywords:
Status: CLOSED UPSTREAM
Alias: None
Product: vdsm
Classification: oVirt
Component: SuperVDSM
Version: 4.50.0.13
Hardware: Unspecified
OS: Linux
unspecified
high
Target Milestone: ---
: ---
Assignee: eraviv
QA Contact: eraviv
URL:
Whiteboard:
Depends On: 2084118 2121315 2148207
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-05-11 15:05 UTC by Sandro Bonazzola
Modified: 2022-11-24 15:50 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-10-05 11:21:23 UTC
oVirt Team: Network
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHV-46011 0 None None None 2022-05-11 15:56:51 UTC

Description Sandro Bonazzola 2022-05-11 15:05:04 UTC
Description of problem:

Deploying on CentOS Stream 9 the network setup is failing with:

Traceback (most recent call last):
    File "/usr/lib/python3.9/site-packages/yajsonrpc/__init__.py", line 349, in _handle_request
    res = method(**params)
    File "/usr/lib/python3.9/site-packages/vdsm/rpc/Bridge.py", line 198, in _dynamicMethod
    result = fn(*methodArgs)
    File "<decorator-gen-506>", line 2, in setupNetworks
    File "/usr/lib/python3.9/site-packages/vdsm/common/api.py", line 50, in method
    ret = func(*args, **kwargs)
    File "/usr/lib/python3.9/site-packages/vdsm/API.py", line 1592, in setupNetworks
    supervdsm.getProxy().setupNetworks(networks, bondings, options)
    File "/usr/lib/python3.9/site-packages/vdsm/common/supervdsm.py", line 56, in __call__
    return callMethod()
    File "/usr/lib/python3.9/site-packages/vdsm/common/supervdsm.py", line 53, in <lambda>
    getattr(self._supervdsmProxy._svdsm, self._funcName)(*args,
    File "<string>", line 2, in setupNetworks
    File "/usr/lib64/python3.9/multiprocessing/managers.py", line 825, in _callmethod
    raise convert_to_error(kind, result)
KeyError: 'bridge'


I also noticed:

May 11 14:40:10 node45el9.lab ovs-vswitchd[26048]: ovs|00037|rconn|ERR|br-int<->unix#3: no response to inactivity probe after 60 seconds, disconnecting

Noticed a few selinux denials reported in bug #2081636.

Applied the changes suggested by SELinux:


ausearch -c 'pk12util' --raw | audit2allow -M my-pk12util
semodule -X 300 -i my-pk12util.pp
ausearch -c 'lldpad' --raw | audit2allow -M my-lldpad
semodule -X 300 -i my-lldpad.pp


Now there are no more selinux denials and ovs seems to be up and running but I still get:

May 11 15:00:41 node45el9.lab vdsm[45919]: ERROR Internal server error
                                           Traceback (most recent call last):
                                             File "/usr/lib/python3.9/site-packages/yajsonrpc/__init__.py", line 349, in _handle_request
                                               res = method(**params)
                                             File "/usr/lib/python3.9/site-packages/vdsm/rpc/Bridge.py", line 198, in _dynamicMethod
                                               result = fn(*methodArgs)
                                             File "<decorator-gen-506>", line 2, in setupNetworks
                                             File "/usr/lib/python3.9/site-packages/vdsm/common/api.py", line 50, in method
                                               ret = func(*args, **kwargs)
                                             File "/usr/lib/python3.9/site-packages/vdsm/API.py", line 1592, in setupNetworks
                                               supervdsm.getProxy().setupNetworks(networks, bondings, options)
                                             File "/usr/lib/python3.9/site-packages/vdsm/common/supervdsm.py", line 56, in __call__
                                               return callMethod()
                                             File "/usr/lib/python3.9/site-packages/vdsm/common/supervdsm.py", line 53, in <lambda>
                                               getattr(self._supervdsmProxy._svdsm, self._funcName)(*args,
                                             File "<string>", line 2, in setupNetworks
                                             File "/usr/lib64/python3.9/multiprocessing/managers.py", line 825, in _callmethod
                                               raise convert_to_error(kind, result)
                                           KeyError: 'ipv4'
May 11 15:00:41 node45el9.lab daemonAdapter[40164]: HAHA Some([BridgeVlanEntry { vid: Some(1), vid_range: None, is_pvid: true, is_egress_untagged: true }])

Comment 1 Sandro Bonazzola 2022-05-11 16:48:52 UTC
Looks like CentOS Stream 9 specific issue as deploy on Alma Linux 9.0 Beta works after executing:
ausearch -c 'lldpad' --raw | audit2allow -M my-lldpad
semodule -X 300 -i my-lldpad.pp
ausearch -c 'modprobe' --raw | audit2allow -M my-modprobe
semodule -X 300 -i my-modprobe.pp

for handling selinux denials reported in the journal.

Enabling oVirt 4.5 on Alma 9 beta:

```
cat >/etc/yum.repos.d/CentOS-Stream-Extras-common.repo <<'EOF'
[c9s-extras-common]
name=CentOS Stream $releasever - Extras packages
metalink=https://mirrors.centos.org/metalink?repo=centos-extras-sig-extras-common-$stream&arch=$basearch&protocol=https,http
gpgkey=https://www.centos.org/keys/RPM-GPG-KEY-CentOS-SIG-Extras
gpgcheck=1
repo_gpgcheck=0
metadata_expire=6h
countme=1
enabled=1
EOF

echo "9-stream" > /etc/yum/vars/stream

dnf distro-sync --nobest

```

going to update the install guide with this once I get it a bit more tested.

Comment 2 Ales Musil 2022-05-16 05:52:12 UTC
Hi, 

the selinux is probably new one, but the  KeyError: 'ipv4' is due to backward incompatibility of nmstate-2.1. 
The reason why it works on Alma is probably that it has nmstate-2.0 which is still fine. 

The nmstate team is aware of that and hopefully the fix will land soon (PR is already posted).

Comment 3 Martin Perina 2022-05-16 06:59:28 UTC
Based on above targeting to 4.5.1, let's see when relevant nmstate update will land in CS9

Comment 9 eraviv 2022-08-22 07:59:34 UTC
fails on a different issue (attached supervdsm log):

Traceback (most recent call last):
  .....
  File "/usr/lib/python3.9/site-packages/vdsm/common/supervdsm.py", line 53, in <lambda>
    getattr(self._supervdsmProxy._svdsm, self._funcName)(*args,
  File "<string>", line 2, in setupNetworks
  File "/usr/lib64/python3.9/multiprocessing/managers.py", line 825, in _callmethod
    raise convert_to_error(kind, result)
libnmstate.error.NmstateInternalError: DbusConnectionError: org.freedesktop.NetworkManager.Settings.Connection.InvalidProperty: connection.slave-type: A connection with a 'bridge-port' setting must have the slave-type set to 'bridge'. Instead it is 'bond' dbus: Some(MethodError("org.freedesktop.NetworkManager.Settings.Connection.InvalidProperty", Some("connection.slave-type: A connection with a 'bridge-port' setting must have the slave-type set to 'bridge'. Instead it is 'bond'"), Msg { type: Error, sender: ":1.6", reply-serial: 332, body: Signature: [
	s (115),
] }))


with:
nmstate-libs-2.2.0-0.alpha.20220810.8e122d43.el9.x86_64
python3-libnmstate-2.2.0-0.alpha.20220810.8e122d43.el9.x86_64

Comment 10 eraviv 2022-08-28 03:18:30 UTC
Issue from comment #9 acked by nmstate team and dedicated bug #2121315 opened to track it. Adding 'depends on' it.

Comment 11 Martin Perina 2022-10-05 11:21:23 UTC
Moving to https://github.com/oVirt/vdsm/issues/338


Note You need to log in before you can comment on or make changes to this bug.