Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2181532

Summary: [Multirhel] container-puppet-ovn_controller failed to start on rhel8 compute node
Product: Red Hat OpenStack Reporter: Khomesh Thakre <kthakre>
Component: puppet-ovnAssignee: Miro Tomaska <mtomaska>
Status: CLOSED ERRATA QA Contact: Bharath M V <bmv>
Severity: urgent Docs Contact:
Priority: high    
Version: 17.1 (Wallaby)CC: arcsingh, bshephar, drosenfe, ekuris, jlibosva, mciecier, mtomaska, ykarel
Target Milestone: betaKeywords: Triaged
Target Release: 17.1Flags: mtomaska: needinfo-
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: puppet-ovn-18.6.1-1.20230413011101.7805f7e.el9ost Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-08-16 01:14:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Khomesh Thakre 2023-03-24 12:27:56 UTC 
Description of problem:
When deploying multirhel, we face an issue with container-puppet-ovn_controller on the rhel8 compute node. 

In log, we see below the traceback

~~~
2023-03-23T15:05:08.509178350+00:00 stdout F include ::tripleo::packages
2023-03-23T15:05:08.509178350+00:00 stdout F include tripleo::profile::base::neutron::agents::ovn
2023-03-23T15:05:08.509178350+00:00 stdout F 
2023-03-23T15:05:08.851214125+00:00 stdout F Running puppet
2023-03-23T15:05:08.851267318+00:00 stderr F + /usr/bin/puppet apply --summarize --detailed-exitcodes --color=false --modulepath=/etc/puppet/modules:/usr/share/openstack-puppet/modules --tags '"file,file_line,concat,augeas,cron,vs_config,exec"' /etc/config.pp
2023-03-23T15:05:08.851267318+00:00 stderr F + logger -s -t puppet-user
2023-03-23T15:05:12.118270302+00:00 stderr F <13>Mar 23 15:05:08 puppet-user: Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5
2023-03-23T15:05:12.118270302+00:00 stderr F <13>Mar 23 15:05:12 puppet-user:    (file: /etc/puppet/hiera.yaml)
2023-03-23T15:05:12.118270302+00:00 stderr F <13>Mar 23 15:05:12 puppet-user: Warning: Undefined variable '::deploy_config_name'; 
2023-03-23T15:05:12.118270302+00:00 stderr F <13>Mar 23 15:05:12 puppet-user:    (file & line not available)
2023-03-23T15:05:12.183028171+00:00 stderr F <13>Mar 23 15:05:12 puppet-user: Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/7.10/deprecated_language.html
2023-03-23T15:05:12.183028171+00:00 stderr F <13>Mar 23 15:05:12 puppet-user:    (file & line not available)
2023-03-23T15:05:12.451714976+00:00 stderr F <13>Mar 23 15:05:12 puppet-user: Notice: Compiled catalog for computerhel8-0.redhat.local in environment production in 0.41 seconds
2023-03-23T15:05:12.572133567+00:00 stderr F <13>Mar 23 15:05:12 puppet-user: Notice: /Stage[main]/Ovn::Controller/Vs_config[external_ids:ovn-remote]/ensure: created
2023-03-23T15:05:12.630244664+00:00 stderr F <13>Mar 23 15:05:12 puppet-user: Notice: /Stage[main]/Ovn::Controller/Vs_config[external_ids:ovn-encap-type]/ensure: created
2023-03-23T15:05:12.685145491+00:00 stderr F <13>Mar 23 15:05:12 puppet-user: Notice: /Stage[main]/Ovn::Controller/Vs_config[external_ids:ovn-encap-ip]/ensure: created
2023-03-23T15:05:12.752055928+00:00 stderr F <13>Mar 23 15:05:12 puppet-user: Notice: /Stage[main]/Ovn::Controller/Vs_config[external_ids:hostname]/value: value changed 'computerhel8-0' to 'computerhel8-0.redhat.local'
2023-03-23T15:05:12.808347839+00:00 stderr F <13>Mar 23 15:05:12 puppet-user: Notice: /Stage[main]/Ovn::Controller/Vs_config[external_ids:ovn-bridge]/ensure: created
2023-03-23T15:05:12.854737455+00:00 stderr F <13>Mar 23 15:05:12 puppet-user: Notice: /Stage[main]/Ovn::Controller/Vs_config[external_ids:ovn-remote-probe-interval]/ensure: created
2023-03-23T15:05:12.900045742+00:00 stderr F <13>Mar 23 15:05:12 puppet-user: Notice: /Stage[main]/Ovn::Controller/Vs_config[external_ids:ovn-openflow-probe-interval]/ensure: created
2023-03-23T15:05:12.934178752+00:00 stderr F <13>Mar 23 15:05:12 puppet-user: Notice: /Stage[main]/Ovn::Controller/Vs_config[external_ids:ovn-monitor-all]/ensure: created
2023-03-23T15:05:12.992319430+00:00 stderr F <13>Mar 23 15:05:12 puppet-user: Notice: /Stage[main]/Ovn::Controller/Vs_config[external_ids:ovn-ofctrl-wait-before-clear]/ensure: created
2023-03-23T15:05:13.057244286+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Notice: /Stage[main]/Ovn::Controller/Vs_config[external_ids:ovn-encap-tos]/ensure: created
2023-03-23T15:05:13.105078349+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Notice: /Stage[main]/Ovn::Controller/Vs_config[external_ids:ovn-chassis-mac-mappings]/ensure: created
2023-03-23T15:05:13.143542231+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Notice: /Stage[main]/Ovn::Controller/Vs_config[external_ids:ovn-bridge-mappings]/ensure: created
2023-03-23T15:05:13.203058356+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Notice: /Stage[main]/Ovn::Controller/Vs_config[external_ids:ovn-match-northd-version]/ensure: created
2023-03-23T15:05:13.232112481+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Error: /Stage[main]/Ovn::Controller/Exec[br-ex]: Could not evaluate: can't create Thread: Operation not permitted
2023-03-23T15:05:13.234044194+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Error: /Stage[main]/Ovn::Controller/Exec[br-tenant]: Could not evaluate: can't create Thread: Operation not permitted
2023-03-23T15:05:13.259067528+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Notice: Applied catalog in 0.77 seconds
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Application:
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user:    Initial environment: production
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user:    Converged environment: production
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user:          Run mode: user
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Changes:
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user:             Total: 13
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Events:
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user:           Success: 13
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user:           Failure: 2
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user:             Total: 15
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Resources:
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user:           Changed: 13
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user:           Skipped: 13
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user:       Out of sync: 15
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user:            Failed: 2
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user:             Total: 30
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Time:
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user:              Exec: 0.01
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user:    Config retrieval: 0.48
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user:         Vs config: 0.67
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user:    Transaction evaluation: 0.74
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user:    Catalog application: 0.77
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user:          Last run: 1679583913
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user:             Total: 0.77
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Version:
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user:            Config: 1679583912
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user:            Puppet: 7.10.0
~~~

After further debugging we find below puppet error. 

~~~
2023-03-23T08:00:58.991850213+00:00 stderr F <13>Mar 23 08:00:58 puppet-user: Debug: Exec[br-ex](provider=posix): Executing check ‘ovs-vsctl get bridge br-ex other-config:mac-table-size | grep -q -w 50000'
2023-03-23T08:00:58.992133129+00:00 stderr F <13>Mar 23 08:00:58 puppet-user: Error: /Stage[main]/Ovn::Controller/Exec[br-ex]: Could not evaluate: can’t create Thread: Operation not permitted
~~~

This is ovn-dvr setup. The issue is not seen in non-dvr topology.


Version-Release number of selected component (if applicable):
RHOS-17.1-RHEL-9-20230315.n.1 puddle was use to deploy.

How reproducible:
100%

Steps to Reproduce:
1. Deploy multirhel with ovn-dvr 
2.
3.
Comment 8 Brendan Shephard 2023-04-12 00:46:20 UTC 
Looks like the same issue for iscsi:
https://bugzilla.redhat.com/show_bug.cgi?id=2185933
Comment 9 Brendan Shephard 2023-04-12 02:58:13 UTC 
*** Bug 2185932 has been marked as a duplicate of this bug. ***
Comment 10 Miro Tomaska 2023-04-14 16:10:36 UTC 
Khomesh,

Looks like the permission problem is resolved with the latest puddle for ovn-controller[1] and iscid[2] BZ2185932. The CI job[3] failed on the later stage for overcloud update but that does not look like its related this this BZ.


[1]http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/staging/DFG-upgrades-updates-17.1-from-passed_phase2-HA_multi_rhel-ipv4/10/computerhel8-0/var/log/containers/stdouts/container-puppet-ovn_controller.log.gz
[2]http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/staging/DFG-upgrades-updates-17.1-from-passed_phase2-HA_multi_rhel-ipv4/10/computerhel8-0/var/log/containers/stdouts/container-puppet-iscsid.log.gz
[3]https://rhos-ci-staging-jenkins.lab.eng.tlv2.redhat.com/view/DFG/view/upgrades/view/update/job/DFG-upgrades-updates-17.1-from-passed_phase2-HA_multi_rhel-ipv4/10/
Comment 11 Khomesh Thakre 2023-04-19 06:06:08 UTC 
Hello Miro,

In the latest run deployment has been completed.

Thanks for the help.
Comment 25 errata-xmlrpc 2023-08-16 01:14:34 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 17.1 (Wallaby)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2023:4577