Description of problem: Podman with netavark is having some issues when running as user root or with sudo. Getting some permission denied error, due to which overcloud deployment fails. (undercloud) [stack@undercloud ~]sudo podman run --rm --user heat --volume /home/stack/overcloud-deploy/overcloud/heat-launcher/heat.conf:/etc/heat/heat.conf:z --volume /home/stack/overcloud-deplploy/overcloud/heat-launcher:/home/stack/overcloud-deploy/overcloud/heat-launcher:z localhost/tripleo/openstack-heat-api:ephemeral --net podman heat-manage db_sync Error: netavark: failed to configure bridge and veth interface: failed while configuring network interface: failed to set ip address to podman0: Permission denied (os error 13) The problem does not occur with podman when we run it without root privileges. But here in this case, podman is unable to pull the image(openstack-heat-api:ephemeral) without privileges. Actual results: 2022-09-01 08:46:46.320 807944 WARNING tripleoclient.utils.safe_write [-] The output file /home/stack/overcloud-deploy/overcloud/overcloud-deployment_status.yaml will be overriden 2022-09-01 08:46:46.320 807944 INFO tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Using ephemeral heat for stack operation 2022-09-01 08:46:46.953 807944 INFO tripleoclient.heat_launcher [-] Skipping container image pull. 2022-09-01 08:46:46.960 807944 INFO tripleoclient.heat_launcher [-] Checking that database is up 2022-09-01 08:46:47.272 807944 INFO tripleoclient.heat_launcher [-] Checking that message bus (rabbitmq) is up Error: netavark: failed to configure bridge and veth interface: failed while configuring network interface: failed to set ip address to podman0: Permission denied (os error 13) 2022-09-01 08:46:49.020 807944 WARNING tripleoclient.utils.safe_write [-] The output file /home/stack/overcloud-deploy/overcloud/overcloud-deployment_status.yaml will be overriden: subprocess.CalledProcessError: Command '['sudo', 'podman', 'run', '--rm', '--user', 'heat', '--volume', '/home/stack/overcloud-deploy/overcloud/heat-launcher/heat.conf:/etc/heat/heat.conf:z', '--volume', '/home/stack/overcloud-deploy/overcloud/heat-launcher:/home/stack/overcloud-deploy/overcloud/heat-launcher:z', 'localhost/tripleo/openstack-heat-api:ephemeral', 'heat-manage', 'db_sync']' returned non-zero exit status 126. 2022-09-01 08:46:49.021 807944 INFO tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Stopping ephemeral heat. 2022-09-01 08:46:49.104 807944 INFO tripleoclient.heat_launcher [-] Pod does not exist: ephemeral-heat 2022-09-01 08:46:49.287 807944 INFO tripleoclient.heat_launcher [-] Starting back up of heat db 2022-09-01 08:46:49.478 807944 INFO tripleoclient.heat_launcher [-] Created tarfile /home/stack/overcloud-deploy/overcloud/heat-launcher/heat-db.sql-1662022006.3209467.tar.bzip2 2022-09-01 08:46:49.478 807944 INFO tripleoclient.heat_launcher [-] Deleting /home/stack/overcloud-deploy/overcloud/heat-launcher/heat-db.sql 2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Exception occured while running the command: subprocess.CalledProcessError: Command '['sudo', 'podman', 'run', '--rm', '--user', 'heat', '--volume', '/home/stack/overcloud-deploy/overcloud/heat-launcher/heat.conf:/etc/heat/heat.conf:z', '--volume', '/home/stack/overcloud-deploy/overcloud/heat-launcher:/home/stack/overcloud-deploy/overcloud/heat-launcher:z', 'localhost/tripleo/openstack-heat-api:ephemeral', 'heat-manage', 'db_sync']' returned non-zero exit status 126. 2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent call last): 2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/command.py", line 32, in run 2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud super(Command, self).run(parsed_args) 2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/osc_lib/command/command.py", line 39, in run 2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return super(Command, self).run(parsed_args) 2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/cliff/command.py", line 186, in run 2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return_code = self.take_action(parsed_args) or 0 2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1262, in take_action 2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud deployment.set_deployment_status( 2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud self.force_reraise() 2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud raise self.value 2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1197, in take_action 2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud self.setup_ephemeral_heat(parsed_args) 2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 663, in setup_ephemeral_heat 2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud self.orchestration_client = utils.launch_heat(self.heat_launcher) 2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/utils.py", line 2752, in launch_heat 2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud launcher.heat_db_sync(restore_db) 2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/heat_launcher.py", line 556, in heat_db_sync 2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud subprocess.check_call(cmd) 2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib64/python3.9/subprocess.py", line 373, in check_call 2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud raise CalledProcessError(retcode, cmd) 2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud subprocess.CalledProcessError: Command '['sudo', 'podman', 'run', '--rm', '--user', 'heat', '--volume', '/home/stack/overcloud-deploy/overcloud/heat-launcher/heat.conf:/etc/heat/heat.conf:z', '--volume', '/home/stack/overcloud-deploy/overcloud/heat-launcher:/home/stack/overcloud-deploy/overcloud/heat-launcher:z', 'localhost/tripleo/openstack-heat-api:ephemeral', 'heat-manage', 'db_sync']' returned non-zero exit status 126. 2022-09-01 08:46:49.934 807944 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 2022-09-01 08:46:49.937 807944 ERROR openstack [-] Command '['sudo', 'podman', 'run', '--rm', '--user', 'heat', '--volume', '/home/stack/overcloud-deploy/overcloud/heat-launcher/heat.conf:/etc/heat/heat.conf:z', '--volume', '/home/stack/overcloud-deploy/overcloud/heat-launcher:/home/stack/overcloud-deploy/overcloud/heat-launcher:z', 'localhost/tripleo/openstack-heat-api:ephemeral', 'heat-manage', 'db_sync']' returned non-zero exit status 126.: subprocess.CalledProcessError: Command '['sudo', 'podman', 'run', '--rm', '--user', 'heat', '--volume', '/home/stack/overcloud-deploy/overcloud/heat-launcher/heat.conf:/etc/heat/heat.conf:z', '--volume', '/home/stack/overcloud-deploy/overcloud/heat-launcher:/home/stack/overcloud-deploy/overcloud/heat-launcher:z', 'localhost/tripleo/openstack-heat-api:ephemeral', 'heat-manage', 'db_sync']' returned non-zero exit status 126. 2022-09-01 08:46:49.937 807944 INFO osc_lib.shell [-] END return value: 1 Expected results: Successful overcloud deployment Additional info: versions: (undercloud) [stack@undercloud ~]$ rpm -qa |grep -i podman ansible-collection-containers-podman-1.9.4-1.el9ost.noarch podman-4.1.1-1.el9_0.x86_64 podman-catatonit-4.1.1-1.el9_0.x86_64 (undercloud) [stack@undercloud ~]$ rpm -qa |grep -i netavark netavark-1.0.1-36.el9_0.x86_64 openstack_puddle: 17.0_20220825.1
I just did an upgrade on my tripleo environment and now have the same issue. But I don't seem to get it on my OSP17 director node. It is deployed from the public beta repos though rather than internal puddles. To workaround this, we can: $ sudo dnf install containernetworking-plugins -y $ sed -i 's/netavark/cni/g' /etc/containers/containers.conf $ sudo reboot That should work around it. But I'm not sure why it's happening for you on this release. My director node is using the same packages and I don't see the issue on OSP17. Just upstream tripleo on CentOS9. Here's the details from my OSP17 director node: [root@17director ~]# rpm -qa | egrep 'netavark|podman' netavark-1.0.1-36.el9_0.x86_64 podman-catatonit-4.1.1-1.el9_0.x86_64 podman-4.1.1-1.el9_0.x86_64 [root@17director ~]# cat /etc/containers/containers.conf [containers] pids_limit = 4096 [engine] events_logger = "journald" runtime = "crun" infra_image = "registry.access.redhat.com/ubi8/pause" [network] network_backend = "netavark"
i'm brining up an osp17 environment to check this one. We switched to netavark with this patch: https://code.engineering.redhat.com/gerrit/c/tripleo-ansible/+/424337
I can't reproduce this one either. Could we see the output of rpm -qa? How did you install 17.0? The puddle I'm using is RHOS-17.0-RHEL-9-20220909.n.0
Output for rpm -qa : https://privatebin.corp.redhat.com/?993278dd7df84d69#HGxgASk5EA4LehNauY8kE1nrHwo8VxVvYpJb16c3GxCD sos report: http://perfscale.perf.lab.eng.bos.redhat.com/pub/rajesh/sos_report/podman/ openstack_puddle: 17.0_20220825.1
Won't make a blocker request for this one: - the podman command as exposed in the initial report can be changed in order to use "--net host" - this issue actually doesn't prevent deploy
Note that the initial command is wrong - the "--net podman" option must be passed before the image: sudo podman run --rm --user heat \ --volume /home/stack/overcloud-deploy/overcloud/heat-launcher/heat.conf:/etc/heat/heat.conf:z \ --volume /home/stack/overcloud-deplploy/overcloud/heat-launcher:/home/stack/overcloud-deploy/overcloud/heat-launcher:z \ --net podman \ localhost/tripleo/openstack-heat-api:ephemeral heat-manage db_sync
All containers start, deploy is successful, and command from description executes without a permission denied error.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform 17.0.1 bug fix and enhancement advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2023:0271