Bug 2123252 - Error: netavark: failed to configure bridge and veth interface: failed while configuring network interface: failed to set ip address to podman0: Permission denied (os error 13)
Summary: Error: netavark: failed to configure bridge and veth interface: failed while ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-tripleoclient
Version: 17.0 (Wallaby)
Hardware: x86_64
OS: Linux
high
high
Target Milestone: z1
: 17.0
Assignee: Rabi Mishra
QA Contact: Joe H. Rahme
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-09-01 08:52 UTC by Rajesh
Modified: 2023-01-25 12:29 UTC (History)
11 users (show)

Fixed In Version: python-tripleoclient-16.5.1-0.20221207110335.23dbe54.el9ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-01-25 12:28:51 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 859257 0 None stable/wallaby: MERGED python-tripleoclient: Use hostNetwork for ephemeral heat pod (I1d01e149e11cdea1589c90df80445af3db359f6e) 2022-12-07 20:35:13 UTC
OpenStack gerrit 859968 0 None stable/wallaby: MERGED python-tripleoclient: Use host network for db_sync (Ifc72293052e8719cd519400c76c548910a741ff3) 2022-12-07 20:35:20 UTC
Red Hat Issue Tracker OSP-18499 0 None None None 2022-09-01 08:55:09 UTC
Red Hat Product Errata RHBA-2023:0271 0 None None None 2023-01-25 12:29:08 UTC

Description Rajesh 2022-09-01 08:52:11 UTC
Description of problem:

    Podman with netavark is having some issues when running as user root or with sudo. Getting some permission denied error, due to which overcloud deployment fails.

(undercloud) [stack@undercloud ~]sudo podman run --rm --user heat --volume /home/stack/overcloud-deploy/overcloud/heat-launcher/heat.conf:/etc/heat/heat.conf:z --volume /home/stack/overcloud-deplploy/overcloud/heat-launcher:/home/stack/overcloud-deploy/overcloud/heat-launcher:z localhost/tripleo/openstack-heat-api:ephemeral --net podman heat-manage db_sync                      
Error: netavark: failed to configure bridge and veth interface: failed while configuring network interface: failed to set ip address to podman0: Permission denied (os error 13)

The problem does not occur with podman when we run it without root privileges. But here in this case, podman is unable to pull the image(openstack-heat-api:ephemeral) without privileges.


Actual results:

2022-09-01 08:46:46.320 807944 WARNING tripleoclient.utils.safe_write [-] The output file /home/stack/overcloud-deploy/overcloud/overcloud-deployment_status.yaml will be overriden
2022-09-01 08:46:46.320 807944 INFO tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Using ephemeral heat for stack operation
2022-09-01 08:46:46.953 807944 INFO tripleoclient.heat_launcher [-] Skipping container image pull.
2022-09-01 08:46:46.960 807944 INFO tripleoclient.heat_launcher [-] Checking that database is up
2022-09-01 08:46:47.272 807944 INFO tripleoclient.heat_launcher [-] Checking that message bus (rabbitmq) is up
Error: netavark: failed to configure bridge and veth interface: failed while configuring network interface: failed to set ip address to podman0: Permission denied (os error 13)
2022-09-01 08:46:49.020 807944 WARNING tripleoclient.utils.safe_write [-] The output file /home/stack/overcloud-deploy/overcloud/overcloud-deployment_status.yaml will be overriden: subprocess.CalledProcessError: Command '['sudo', 'podman', 'run', '--rm', '--user', 'heat', '--volume', '/home/stack/overcloud-deploy/overcloud/heat-launcher/heat.conf:/etc/heat/heat.conf:z', '--volume', '/home/stack/overcloud-deploy/overcloud/heat-launcher:/home/stack/overcloud-deploy/overcloud/heat-launcher:z', 'localhost/tripleo/openstack-heat-api:ephemeral', 'heat-manage', 'db_sync']' returned non-zero exit status 126.
2022-09-01 08:46:49.021 807944 INFO tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Stopping ephemeral heat.
2022-09-01 08:46:49.104 807944 INFO tripleoclient.heat_launcher [-] Pod does not exist: ephemeral-heat
2022-09-01 08:46:49.287 807944 INFO tripleoclient.heat_launcher [-] Starting back up of heat db
2022-09-01 08:46:49.478 807944 INFO tripleoclient.heat_launcher [-] Created tarfile /home/stack/overcloud-deploy/overcloud/heat-launcher/heat-db.sql-1662022006.3209467.tar.bzip2
2022-09-01 08:46:49.478 807944 INFO tripleoclient.heat_launcher [-] Deleting /home/stack/overcloud-deploy/overcloud/heat-launcher/heat-db.sql
2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Exception occured while running the command: subprocess.CalledProcessError: Command '['sudo', 'podman', 'run', '--rm', '--user', 'heat', '--volume', '/home/stack/overcloud-deploy/overcloud/heat-launcher/heat.conf:/etc/heat/heat.conf:z', '--volume', '/home/stack/overcloud-deploy/overcloud/heat-launcher:/home/stack/overcloud-deploy/overcloud/heat-launcher:z', 'localhost/tripleo/openstack-heat-api:ephemeral', 'heat-manage', 'db_sync']' returned non-zero exit status 126.
2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent call last):
2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/tripleoclient/command.py", line 32, in run
2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     super(Command, self).run(parsed_args)
2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/osc_lib/command/command.py", line 39, in run
2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     return super(Command, self).run(parsed_args)
2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/cliff/command.py", line 186, in run
2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     return_code = self.take_action(parsed_args) or 0
2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1262, in take_action
2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     deployment.set_deployment_status(
2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     self.force_reraise()
2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     raise self.value
2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1197, in take_action
2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     self.setup_ephemeral_heat(parsed_args)
2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 663, in setup_ephemeral_heat
2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     self.orchestration_client = utils.launch_heat(self.heat_launcher)
2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/tripleoclient/utils.py", line 2752, in launch_heat
2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     launcher.heat_db_sync(restore_db)
2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/tripleoclient/heat_launcher.py", line 556, in heat_db_sync
2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     subprocess.check_call(cmd)
2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib64/python3.9/subprocess.py", line 373, in check_call
2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     raise CalledProcessError(retcode, cmd)
2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud subprocess.CalledProcessError: Command '['sudo', 'podman', 'run', '--rm', '--user', 'heat', '--volume', '/home/stack/overcloud-deploy/overcloud/heat-launcher/heat.conf:/etc/heat/heat.conf:z', '--volume', '/home/stack/overcloud-deploy/overcloud/heat-launcher:/home/stack/overcloud-deploy/overcloud/heat-launcher:z', 'localhost/tripleo/openstack-heat-api:ephemeral', 'heat-manage', 'db_sync']' returned non-zero exit status 126.
2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-09-01 08:46:49.937 807944 ERROR openstack [-] Command '['sudo', 'podman', 'run', '--rm', '--user', 'heat', '--volume', '/home/stack/overcloud-deploy/overcloud/heat-launcher/heat.conf:/etc/heat/heat.conf:z', '--volume', '/home/stack/overcloud-deploy/overcloud/heat-launcher:/home/stack/overcloud-deploy/overcloud/heat-launcher:z', 'localhost/tripleo/openstack-heat-api:ephemeral', 'heat-manage', 'db_sync']' returned non-zero exit status 126.: subprocess.CalledProcessError: Command '['sudo', 'podman', 'run', '--rm', '--user', 'heat', '--volume', '/home/stack/overcloud-deploy/overcloud/heat-launcher/heat.conf:/etc/heat/heat.conf:z', '--volume', '/home/stack/overcloud-deploy/overcloud/heat-launcher:/home/stack/overcloud-deploy/overcloud/heat-launcher:z', 'localhost/tripleo/openstack-heat-api:ephemeral', 'heat-manage', 'db_sync']' returned non-zero exit status 126.
2022-09-01 08:46:49.937 807944 INFO osc_lib.shell [-] END return value: 1


Expected results:
Successful overcloud deployment

Additional info:

versions: 
(undercloud) [stack@undercloud ~]$ rpm -qa |grep -i podman
ansible-collection-containers-podman-1.9.4-1.el9ost.noarch
podman-4.1.1-1.el9_0.x86_64
podman-catatonit-4.1.1-1.el9_0.x86_64

(undercloud) [stack@undercloud ~]$ rpm -qa |grep -i netavark
netavark-1.0.1-36.el9_0.x86_64

openstack_puddle: 17.0_20220825.1

Comment 2 Brendan Shephard 2022-09-07 01:19:26 UTC
I just did an upgrade on my tripleo environment and now have the same issue. But I don't seem to get it on my OSP17 director node. It is deployed from the public beta repos though rather than internal puddles. 

To workaround this, we can:
$ sudo dnf install containernetworking-plugins -y
$ sed -i 's/netavark/cni/g' /etc/containers/containers.conf
$ sudo reboot

That should work around it. But I'm not sure why it's happening for you on this release. My director node is using the same packages and I don't see the issue on OSP17. Just upstream tripleo on CentOS9.

Here's the details from my OSP17 director node:
[root@17director ~]# rpm -qa | egrep 'netavark|podman'
netavark-1.0.1-36.el9_0.x86_64
podman-catatonit-4.1.1-1.el9_0.x86_64
podman-4.1.1-1.el9_0.x86_64


[root@17director ~]# cat /etc/containers/containers.conf

[containers]
pids_limit = 4096
[engine]
events_logger = "journald"
runtime = "crun"
infra_image = "registry.access.redhat.com/ubi8/pause"
[network]
network_backend = "netavark"

Comment 3 James Slagle 2022-09-12 19:09:54 UTC
i'm brining up an osp17 environment to check this one. We switched to netavark with this patch:
https://code.engineering.redhat.com/gerrit/c/tripleo-ansible/+/424337

Comment 4 James Slagle 2022-09-12 19:57:01 UTC
I can't reproduce this one either. Could we see the output of rpm -qa? How did you install 17.0?
The puddle I'm using is RHOS-17.0-RHEL-9-20220909.n.0

Comment 14 Cédric Jeanneret 2022-12-07 10:51:26 UTC
Won't make a blocker request for this one:
- the podman command as exposed in the initial report can be changed in order to use "--net host"
- this issue actually doesn't prevent deploy

Comment 20 Cédric Jeanneret 2022-12-15 08:15:51 UTC
Note that the initial command is wrong - the "--net podman" option must be passed before the image:

sudo podman run --rm --user heat \
 --volume /home/stack/overcloud-deploy/overcloud/heat-launcher/heat.conf:/etc/heat/heat.conf:z \
 --volume /home/stack/overcloud-deplploy/overcloud/heat-launcher:/home/stack/overcloud-deploy/overcloud/heat-launcher:z \
 --net podman \
 localhost/tripleo/openstack-heat-api:ephemeral heat-manage db_sync

Comment 21 David Rosenfeld 2022-12-19 15:39:39 UTC
All containers start, deploy is successful, and command from description executes without a permission denied error.

Comment 28 errata-xmlrpc 2023-01-25 12:28:51 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 17.0.1 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:0271


Note You need to log in before you can comment on or make changes to this bug.