Bug 2181532 - [Multirhel] container-puppet-ovn_controller failed to start on rhel8 compute node
Summary: [Multirhel] container-puppet-ovn_controller failed to start on rhel8 compute ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: puppet-ovn
Version: 17.1 (Wallaby)
Hardware: Unspecified
OS: Unspecified
high
urgent
Target Milestone: beta
: 17.1
Assignee: Miro Tomaska
QA Contact: Bharath M V
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-03-24 12:27 UTC by Khomesh Thakre
Modified: 2023-08-16 01:15 UTC (History)
8 users (show)

Fixed In Version: puppet-ovn-18.6.1-1.20230413011101.7805f7e.el9ost
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-08-16 01:14:34 UTC
Target Upstream Version:
Embargoed:
mtomaska: needinfo-


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 879527 0 None MERGED [Stable Only] Handle string value for ovn_bridge_mappings 2023-04-12 04:01:04 UTC
OpenStack gerrit 879861 0 None MERGED [Stable Only] Handle string value for ovn_bridge_mappings 2023-04-12 04:03:01 UTC
OpenStack gerrit 879862 0 None MERGED [Stable Only] Handle string value for ovn_bridge_mappings 2023-04-12 04:03:01 UTC
Red Hat Issue Tracker OSP-23632 0 None None None 2023-03-24 12:29:42 UTC
Red Hat Product Errata RHEA-2023:4577 0 None None None 2023-08-16 01:15:11 UTC

Description Khomesh Thakre 2023-03-24 12:27:56 UTC
Description of problem:
When deploying multirhel, we face an issue with container-puppet-ovn_controller on the rhel8 compute node. 

In log, we see below the traceback

~~~
2023-03-23T15:05:08.509178350+00:00 stdout F include ::tripleo::packages
2023-03-23T15:05:08.509178350+00:00 stdout F include tripleo::profile::base::neutron::agents::ovn
2023-03-23T15:05:08.509178350+00:00 stdout F 
2023-03-23T15:05:08.851214125+00:00 stdout F Running puppet
2023-03-23T15:05:08.851267318+00:00 stderr F + /usr/bin/puppet apply --summarize --detailed-exitcodes --color=false --modulepath=/etc/puppet/modules:/usr/share/openstack-puppet/modules --tags '"file,file_line,concat,augeas,cron,vs_config,exec"' /etc/config.pp
2023-03-23T15:05:08.851267318+00:00 stderr F + logger -s -t puppet-user
2023-03-23T15:05:12.118270302+00:00 stderr F <13>Mar 23 15:05:08 puppet-user: Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5
2023-03-23T15:05:12.118270302+00:00 stderr F <13>Mar 23 15:05:12 puppet-user:    (file: /etc/puppet/hiera.yaml)
2023-03-23T15:05:12.118270302+00:00 stderr F <13>Mar 23 15:05:12 puppet-user: Warning: Undefined variable '::deploy_config_name'; 
2023-03-23T15:05:12.118270302+00:00 stderr F <13>Mar 23 15:05:12 puppet-user:    (file & line not available)
2023-03-23T15:05:12.183028171+00:00 stderr F <13>Mar 23 15:05:12 puppet-user: Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/7.10/deprecated_language.html
2023-03-23T15:05:12.183028171+00:00 stderr F <13>Mar 23 15:05:12 puppet-user:    (file & line not available)
2023-03-23T15:05:12.451714976+00:00 stderr F <13>Mar 23 15:05:12 puppet-user: Notice: Compiled catalog for computerhel8-0.redhat.local in environment production in 0.41 seconds
2023-03-23T15:05:12.572133567+00:00 stderr F <13>Mar 23 15:05:12 puppet-user: Notice: /Stage[main]/Ovn::Controller/Vs_config[external_ids:ovn-remote]/ensure: created
2023-03-23T15:05:12.630244664+00:00 stderr F <13>Mar 23 15:05:12 puppet-user: Notice: /Stage[main]/Ovn::Controller/Vs_config[external_ids:ovn-encap-type]/ensure: created
2023-03-23T15:05:12.685145491+00:00 stderr F <13>Mar 23 15:05:12 puppet-user: Notice: /Stage[main]/Ovn::Controller/Vs_config[external_ids:ovn-encap-ip]/ensure: created
2023-03-23T15:05:12.752055928+00:00 stderr F <13>Mar 23 15:05:12 puppet-user: Notice: /Stage[main]/Ovn::Controller/Vs_config[external_ids:hostname]/value: value changed 'computerhel8-0' to 'computerhel8-0.redhat.local'
2023-03-23T15:05:12.808347839+00:00 stderr F <13>Mar 23 15:05:12 puppet-user: Notice: /Stage[main]/Ovn::Controller/Vs_config[external_ids:ovn-bridge]/ensure: created
2023-03-23T15:05:12.854737455+00:00 stderr F <13>Mar 23 15:05:12 puppet-user: Notice: /Stage[main]/Ovn::Controller/Vs_config[external_ids:ovn-remote-probe-interval]/ensure: created
2023-03-23T15:05:12.900045742+00:00 stderr F <13>Mar 23 15:05:12 puppet-user: Notice: /Stage[main]/Ovn::Controller/Vs_config[external_ids:ovn-openflow-probe-interval]/ensure: created
2023-03-23T15:05:12.934178752+00:00 stderr F <13>Mar 23 15:05:12 puppet-user: Notice: /Stage[main]/Ovn::Controller/Vs_config[external_ids:ovn-monitor-all]/ensure: created
2023-03-23T15:05:12.992319430+00:00 stderr F <13>Mar 23 15:05:12 puppet-user: Notice: /Stage[main]/Ovn::Controller/Vs_config[external_ids:ovn-ofctrl-wait-before-clear]/ensure: created
2023-03-23T15:05:13.057244286+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Notice: /Stage[main]/Ovn::Controller/Vs_config[external_ids:ovn-encap-tos]/ensure: created
2023-03-23T15:05:13.105078349+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Notice: /Stage[main]/Ovn::Controller/Vs_config[external_ids:ovn-chassis-mac-mappings]/ensure: created
2023-03-23T15:05:13.143542231+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Notice: /Stage[main]/Ovn::Controller/Vs_config[external_ids:ovn-bridge-mappings]/ensure: created
2023-03-23T15:05:13.203058356+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Notice: /Stage[main]/Ovn::Controller/Vs_config[external_ids:ovn-match-northd-version]/ensure: created
2023-03-23T15:05:13.232112481+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Error: /Stage[main]/Ovn::Controller/Exec[br-ex]: Could not evaluate: can't create Thread: Operation not permitted
2023-03-23T15:05:13.234044194+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Error: /Stage[main]/Ovn::Controller/Exec[br-tenant]: Could not evaluate: can't create Thread: Operation not permitted
2023-03-23T15:05:13.259067528+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Notice: Applied catalog in 0.77 seconds
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Application:
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user:    Initial environment: production
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user:    Converged environment: production
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user:          Run mode: user
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Changes:
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user:             Total: 13
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Events:
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user:           Success: 13
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user:           Failure: 2
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user:             Total: 15
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Resources:
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user:           Changed: 13
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user:           Skipped: 13
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user:       Out of sync: 15
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user:            Failed: 2
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user:             Total: 30
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Time:
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user:              Exec: 0.01
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user:    Config retrieval: 0.48
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user:         Vs config: 0.67
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user:    Transaction evaluation: 0.74
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user:    Catalog application: 0.77
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user:          Last run: 1679583913
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user:             Total: 0.77
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user: Version:
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user:            Config: 1679583912
2023-03-23T15:05:13.262863427+00:00 stderr F <13>Mar 23 15:05:13 puppet-user:            Puppet: 7.10.0
~~~

After further debugging we find below puppet error. 

~~~
2023-03-23T08:00:58.991850213+00:00 stderr F <13>Mar 23 08:00:58 puppet-user: Debug: Exec[br-ex](provider=posix): Executing check ‘ovs-vsctl get bridge br-ex other-config:mac-table-size | grep -q -w 50000'
2023-03-23T08:00:58.992133129+00:00 stderr F <13>Mar 23 08:00:58 puppet-user: Error: /Stage[main]/Ovn::Controller/Exec[br-ex]: Could not evaluate: can’t create Thread: Operation not permitted
~~~

This is ovn-dvr setup. The issue is not seen in non-dvr topology.


Version-Release number of selected component (if applicable):
RHOS-17.1-RHEL-9-20230315.n.1 puddle was use to deploy.

How reproducible:
100%

Steps to Reproduce:
1. Deploy multirhel with ovn-dvr 
2.
3.

Comment 8 Brendan Shephard 2023-04-12 00:46:20 UTC
Looks like the same issue for iscsi:
https://bugzilla.redhat.com/show_bug.cgi?id=2185933

Comment 9 Brendan Shephard 2023-04-12 02:58:13 UTC
*** Bug 2185932 has been marked as a duplicate of this bug. ***

Comment 11 Khomesh Thakre 2023-04-19 06:06:08 UTC
Hello Miro,

In the latest run deployment has been completed.

Thanks for the help.

Comment 25 errata-xmlrpc 2023-08-16 01:14:34 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 17.1 (Wallaby)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2023:4577


Note You need to log in before you can comment on or make changes to this bug.