Description of problem: When deploying multirhel, we face an issue with container-puppet-ovn_controller on the rhel8 compute node. In log, we see below the traceback ~~~ 2023-03-23T15:05:08.509178350+00:00 stdout F include ::tripleo::packages 2023-03-23T15:05:08.509178350+00:00 stdout F include tripleo::profile::base::neutron::agents::ovn 2023-03-23T15:05:08.509178350+00:00 stdout F 2023-03-23T15:05:08.851214125+00:00 stdout F Running puppet 2023-03-23T15:05:08.851267318+00:00 stderr F + /usr/bin/puppet apply --summarize --detailed-exitcodes --color=false --modulepath=/etc/puppet/modules:/usr/share/openstack-puppet/modules --tags '"file,file_line,concat,augeas,cron,vs_config,exec"' /etc/config.pp 2023-03-23T15:05:08.851267318+00:00 stderr F + logger -s -t puppet-user 2023-03-23T15:05:12.118270302+00:00 stderr F <13>Mar 23 15:05:08 puppet-user: Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5 2023-03-23T15:05:12.118270302+00:00 stderr F <13>Mar 23 15:05:12 puppet-user: (file: /etc/puppet/hiera.yaml) 2023-03-23T15:05:12.118270302+00:00 stderr F <13>Mar 23 15:05:12 puppet-user: Warning: Undefined variable '::deploy_config_name'; 2023-03-23T15:05:12.118270302+00:00 stderr F <13>Mar 23 15:05:12 puppet-user: (file & line not available) 2023-03-23T15:05:12.183028171+00:00 stderr F <13>Mar 23 15:05:12 puppet-user: Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/7.10/deprecated_language.html 2023-03-23T15:05:12.183028171+00:00 stderr F <13>Mar 23 15:05:12 puppet-user: (file & line not available) 2023-03-23T15:05:12.451714976+00:00 stderr F <13>Mar 23 15:05:12 puppet-user: Notice: Compiled catalog for computerhel8-0.redhat.local in environment production in 0.41 seconds 2023-03-23T15:05:12.572133567+00:00 stderr F <13>Mar 23 15:05:12 puppet-user: Notice: /Stage[main]/Ovn::Controller/Vs_config[external_ids:ovn-remote]/ensure: created 2023-03-23T15:05:12.630244664+00:00 stderr F <13>Mar 23 15:05:12 puppet-user: Notice: /Stage[main]/Ovn::Controller/Vs_config[external_ids:ovn-encap-type]/ensure: created 2023-03-23T15:05:12.685145491+00:00 stderr F <13>Mar 23 15:05:12 puppet-user: Notice: /Stage[main]/Ovn::Controller/Vs_config[external_ids:ovn-encap-ip]/ensure: created 2023-03-23T15:05:12.752055928+00:00 stderr F <13>Mar 23 15:05:12 puppet-user: Notice: /Stage[main]/Ovn::Controller/Vs_config[external_ids:hostname]/value: value changed 'computerhel8-0' to 'computerhel8-0.redhat.local' 2023-03-23T15:05:12.808347839+00:00 stderr F <13>Mar 23 15:05:12 puppet-user: Notice: /Stage[main]/Ovn::Controller/Vs_config[external_ids:ovn-bridge]/ensure: created 2023-03-23T15:05:12.854737455+00:00 stderr F <13>Mar 23 15:05:12 puppet-user: Notice: /Stage[main]/Ovn::Controller/Vs_config[external_ids:ovn-remote-probe-interval]/ensure: created 2023-03-23T15:05:12.900045742+00:00 stderr F <13>Mar 23 15:05:12 puppet-user: Notice: /Stage[main]/Ovn::Controller/Vs_config[external_ids:ovn-openflow-probe-interval]/ensure: created 2023-03-23T15:05:12.934178752+00:00 stderr F <13>Mar 23 15:05:12 puppet-user: Notice: /Stage[main]/Ovn::Controller/Vs_config[external_ids:ovn-monitor-all]/ensure: created 2023-03-23T15:05:12.992319430+00:00 stderr F <13>Mar 23 15:05:12 puppet-user: Notice: /Stage[main]/Ovn::Controller/Vs_config[external_ids:ovn-ofctrl-wait-before-clear]/ensure: created 2023-03-23T15:05:13.057244286+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Notice: /Stage[main]/Ovn::Controller/Vs_config[external_ids:ovn-encap-tos]/ensure: created 2023-03-23T15:05:13.105078349+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Notice: /Stage[main]/Ovn::Controller/Vs_config[external_ids:ovn-chassis-mac-mappings]/ensure: created 2023-03-23T15:05:13.143542231+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Notice: /Stage[main]/Ovn::Controller/Vs_config[external_ids:ovn-bridge-mappings]/ensure: created 2023-03-23T15:05:13.203058356+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Notice: /Stage[main]/Ovn::Controller/Vs_config[external_ids:ovn-match-northd-version]/ensure: created 2023-03-23T15:05:13.232112481+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Error: /Stage[main]/Ovn::Controller/Exec[br-ex]: Could not evaluate: can't create Thread: Operation not permitted 2023-03-23T15:05:13.234044194+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Error: /Stage[main]/Ovn::Controller/Exec[br-tenant]: Could not evaluate: can't create Thread: Operation not permitted 2023-03-23T15:05:13.259067528+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Notice: Applied catalog in 0.77 seconds 2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Application: 2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Initial environment: production 2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Converged environment: production 2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Run mode: user 2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Changes: 2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Total: 13 2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Events: 2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Success: 13 2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Failure: 2 2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Total: 15 2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Resources: 2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Changed: 13 2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Skipped: 13 2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Out of sync: 15 2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Failed: 2 2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Total: 30 2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Time: 2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Exec: 0.01 2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Config retrieval: 0.48 2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Vs config: 0.67 2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Transaction evaluation: 0.74 2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Catalog application: 0.77 2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Last run: 1679583913 2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Total: 0.77 2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Version: 2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Config: 1679583912 2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Puppet: 7.10.0 ~~~ After further debugging we find below puppet error. ~~~ 2023-03-23T08:00:58.991850213+00:00 stderr F <13>Mar 23 08:00:58 puppet-user: Debug: Exec[br-ex](provider=posix): Executing check ‘ovs-vsctl get bridge br-ex other-config:mac-table-size | grep -q -w 50000' 2023-03-23T08:00:58.992133129+00:00 stderr F <13>Mar 23 08:00:58 puppet-user: Error: /Stage[main]/Ovn::Controller/Exec[br-ex]: Could not evaluate: can’t create Thread: Operation not permitted ~~~ This is ovn-dvr setup. The issue is not seen in non-dvr topology. Version-Release number of selected component (if applicable): RHOS-17.1-RHEL-9-20230315.n.1 puddle was use to deploy. How reproducible: 100% Steps to Reproduce: 1. Deploy multirhel with ovn-dvr 2. 3.
Looks like the same issue for iscsi: https://bugzilla.redhat.com/show_bug.cgi?id=2185933
*** Bug 2185932 has been marked as a duplicate of this bug. ***
Khomesh, Looks like the permission problem is resolved with the latest puddle for ovn-controller[1] and iscid[2] BZ2185932. The CI job[3] failed on the later stage for overcloud update but that does not look like its related this this BZ. [1]http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/staging/DFG-upgrades-updates-17.1-from-passed_phase2-HA_multi_rhel-ipv4/10/computerhel8-0/var/log/containers/stdouts/container-puppet-ovn_controller.log.gz [2]http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/staging/DFG-upgrades-updates-17.1-from-passed_phase2-HA_multi_rhel-ipv4/10/computerhel8-0/var/log/containers/stdouts/container-puppet-iscsid.log.gz [3]https://rhos-ci-staging-jenkins.lab.eng.tlv2.redhat.com/view/DFG/view/upgrades/view/update/job/DFG-upgrades-updates-17.1-from-passed_phase2-HA_multi_rhel-ipv4/10/
Hello Miro, In the latest run deployment has been completed. Thanks for the help.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Release of components for Red Hat OpenStack Platform 17.1 (Wallaby)), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2023:4577