Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2185933

Summary: [Multirhel] container-puppet-iscsid failed to start during deployment
Product: Red Hat OpenStack Reporter: Khomesh Thakre <kthakre>
Component: puppet-tripleoAssignee: OSP Team <rhos-maint>
Status: CLOSED NEXTRELEASE QA Contact: David Rosenfeld <drosenfe>
Severity: low Docs Contact:
Priority: unspecified    
Version: 17.1 (Wallaby)CC: arcsingh, bshephar, camorris, jjoyce, jparker, jschluet, slinaber, tkajinam, tvignaud
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-04-15 02:03:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Khomesh Thakre 2023-04-11 16:28:04 UTC 
Description of problem:

When deploying multirhel, we face an issue with container-puppet-iscsid on the rhel8 compute node with the below error 

~~~
[root@computerhel8-0 tripleo-admin]# cat /var/log/containers/stdouts/container-puppet-iscsid.log
2023-04-11T16:07:42.304883405+00:00 stdout F include ::tripleo::packages
2023-04-11T16:07:42.304883405+00:00 stdout F include tripleo::profile::base::iscsid
2023-04-11T16:07:42.304883405+00:00 stdout F
2023-04-11T16:07:42.544647254+00:00 stdout F Running puppet
2023-04-11T16:07:42.545296088+00:00 stderr F + /usr/bin/puppet apply --summarize --detailed-exitcodes --color=false --modulepath=/etc/puppet/modules:/usr/share/openstack-puppet/modules --tags '"file,file_line,concat,augeas,cron,iscsid_config"' /etc/config.pp
2023-04-11T16:07:42.545336524+00:00 stderr F + logger -s -t puppet-user
2023-04-11T16:07:45.239678198+00:00 stderr F <13>Apr 11 16:07:42 puppet-user: Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5
2023-04-11T16:07:45.239678198+00:00 stderr F <13>Apr 11 16:07:45 puppet-user:    (file: /etc/puppet/hiera.yaml)
2023-04-11T16:07:45.240409375+00:00 stderr F <13>Apr 11 16:07:45 puppet-user: Warning: Undefined variable '::deploy_config_name';
2023-04-11T16:07:45.240409375+00:00 stderr F <13>Apr 11 16:07:45 puppet-user:    (file & line not available)
2023-04-11T16:07:45.279905069+00:00 stderr F <13>Apr 11 16:07:45 puppet-user: Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/7.10/deprecated_language.html
2023-04-11T16:07:45.279905069+00:00 stderr F <13>Apr 11 16:07:45 puppet-user:    (file & line not available)
2023-04-11T16:07:45.353352349+00:00 stderr F <13>Apr 11 16:07:45 puppet-user: Notice: Compiled catalog for computerhel8-0.redhat.local in environment production in 0.14 seconds
2023-04-11T16:07:45.402566864+00:00 stderr F <13>Apr 11 16:07:45 puppet-user: Error: /Stage[main]/Tripleo::Profile::Base::Iscsid/Exec[sync-iqn-from-host]: Could not evaluate: can't create Thread: Operation not permitted
2023-04-11T16:07:45.402938247+00:00 stderr F <13>Apr 11 16:07:45 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Iscsid/Exec[reset-iscsi-initiator-name]: Dependency Exec[sync-iqn-from-host] has failures: true
2023-04-11T16:07:45.403090635+00:00 stderr F <13>Apr 11 16:07:45 puppet-user: Warning: /Stage[main]/Tripleo::Profile::Base::Iscsid/Exec[reset-iscsi-initiator-name]: Skipping because of failed dependencies
2023-04-11T16:07:45.403486558+00:00 stderr F <13>Apr 11 16:07:45 puppet-user: Warning: /Stage[main]/Tripleo::Profile::Base::Iscsid/File[/etc/iscsi/.initiator_reset]: Skipping because of failed dependencies
2023-04-11T16:07:45.403785612+00:00 stderr F <13>Apr 11 16:07:45 puppet-user: Warning: /Stage[main]/Tripleo::Profile::Base::Iscsid/Exec[sync-iqn-to-host]: Skipping because of failed dependencies
2023-04-11T16:07:46.130012116+00:00 stderr F <13>Apr 11 16:07:46 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Iscsid/Augeas[chap_algs in /etc/iscsi/iscsid.conf]/returns: executed successfully
2023-04-11T16:07:46.132968774+00:00 stderr F <13>Apr 11 16:07:46 puppet-user: Notice: Applied catalog in 0.74 seconds
2023-04-11T16:07:46.136235852+00:00 stderr F <13>Apr 11 16:07:46 puppet-user: Application:
2023-04-11T16:07:46.136235852+00:00 stderr F <13>Apr 11 16:07:46 puppet-user:    Initial environment: production
2023-04-11T16:07:46.136235852+00:00 stderr F <13>Apr 11 16:07:46 puppet-user:    Converged environment: production
2023-04-11T16:07:46.136235852+00:00 stderr F <13>Apr 11 16:07:46 puppet-user:          Run mode: user
2023-04-11T16:07:46.136235852+00:00 stderr F <13>Apr 11 16:07:46 puppet-user: Changes:
2023-04-11T16:07:46.136235852+00:00 stderr F <13>Apr 11 16:07:46 puppet-user:             Total: 1
2023-04-11T16:07:46.136235852+00:00 stderr F <13>Apr 11 16:07:46 puppet-user: Events:
2023-04-11T16:07:46.136235852+00:00 stderr F <13>Apr 11 16:07:46 puppet-user:           Failure: 1
2023-04-11T16:07:46.136235852+00:00 stderr F <13>Apr 11 16:07:46 puppet-user:           Success: 1
2023-04-11T16:07:46.136235852+00:00 stderr F <13>Apr 11 16:07:46 puppet-user:             Total: 2
~~~

After enabling debug 

~~~
2023-04-11T08:49:11.028255318+00:00 stderr F <13>Apr 11 08:49:11 puppet-user: Debug: /Package[iscsi-initiator-utils]: Resource is being skipped, unscheduling all events
2023-04-11T08:49:11.028562968+00:00 stderr F <13>Apr 11 08:49:11 puppet-user: Debug: Exec[sync-iqn-from-host](provider=posix): Executing check '/usr/bin/test -f /tmp/iscsi.host/.initiator_reset'
2023-04-11T08:49:11.028758301+00:00 stderr F <13>Apr 11 08:49:11 puppet-user: Error: /Stage[main]/Tripleo::Profile::Base::Iscsid/Exec[sync-iqn-from-host]: Could not evaluate: can't create Thread: Operation not permitted
2023-04-11T08:49:11.029019023+00:00 stderr F <13>Apr 11 08:49:11 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Iscsid/Exec[reset-iscsi-initiator-name]: Dependency Exec[sync-iqn-from-host] has failures: true
2023-04-11T08:49:11.029019023+00:00 stderr F <13>Apr 11 08:49:11 puppet-user: Warning: /Stage[main]/Tripleo::Profile::Base::Iscsid/Exec[reset-iscsi-initiator-name]: Skipping because of failed dependencies
~~~

Version-Release number of selected component (if applicable):
Puddle: RHOS-17.1-RHEL-9-20230404.n.1


How reproducible:
100%

Steps to Reproduce:
1. Deploy multirhel topology
2.
3.

Actual results:


Expected results:


Additional info:
Comment 4 Brendan Shephard 2023-04-12 00:45:48 UTC 
I think we can close these as duplicates of this bug:
https://bugzilla.redhat.com/show_bug.cgi?id=2181532

The problem is that module tries to use exec but doesn't have permissions. So the same fix for the OVN issue will apply here. The updated version of container-tools.
Comment 5 Brendan Shephard 2023-04-12 03:04:23 UTC 
*** Bug 2185932 has been marked as a duplicate of this bug. ***
Comment 6 Khomesh Thakre 2023-04-12 05:32:39 UTC 
Earlier I thought the same, But I have container-tools:rhel8 module enabled on the node.

~~~
[root@computerhel8-0 ~]# dnf module list --enabled

This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.

Last metadata expiration check: 11:20:39 ago on Tue 11 Apr 2023 05:10:52 AM UTC.
rhel8 appstream
Name               Stream        Profiles Summary
container-tools    rhel8 [d][e]  common [ Most recent (rolling) versions of podman, buildah, skopeo, runc, conmon, runc, conmon, CRIU, Udica, etc as well as dependencies such as container-selinux built and tested together, and updated
                                 d]        as frequently as every 12 weeks.
idm                client [d][e] common [ RHEL IdM long term support client module
                                 d]
perl               5.26 [d][e]   common [ Practical Extraction and Report Language
                                 d], mini
                                 mal
perl-IO-Socket-SSL 2.066 [d][e]  common [ Perl library for transparent TLS
                                 d]
perl-libwww-perl   6.34 [d][e]   common [ A Perl interface to the World-Wide Web
                                 d]
python36           3.6 [d][e]    build, c Python programming language, version 3.6
                                 ommon [d
                                 ] [i]
ruby               2.5 [d][e]    common [ An interpreter of object-oriented scripting language
                                 d]
virt               rhel [d][e]   common [ Virtualization module
                                 d]

Hint: [d]efault, [e]nabled, [x]disabled, [i]installed


[root@computerhel8-0 ~]# yum repolist

This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.

repo id                                                                                                  repo name
rhelosp-17.1                                                                                             RHOS-17.1
rhelosp-17.1-fdp-pending                                                                                 Red Hat Enterprise Linux Fast Datapath 8 - x86_64 Nightly
rhelosp-17.1-image-build-override                                                                        RHOS 17.1 Override
rhelosp-ceph-5-tools                                                                                     Ceph 5 Tools
rhos-release                                                                                             RHOS Release
rhos-release-brew                                                                                        RHOS Release Brew
rhos-release-extras                                                                                      RHOS Release Extras
rhosp-ansible-2.9                                                                                        Red Hat Enterprise Linux Ansible 8 - x86_64
rhosp-rhel-8.4-appstream                                                                                 rhel8 appstream
rhosp-rhel-8.4-av                                                                                        Red Hat Enterprise Linux Advanced Virtualization 8 - x86_64
rhosp-rhel-8.4-baseos                                                                                    rhel8 baseos
rhosp-rhel-8.4-fdp                                                                                       Red Hat Enterprise Linux Fast Datapath 8 - x86_64
rhosp-rhel-8.4-ha                                                                                        Red Hat Enterprise Linux 8 - x86_64 - HA
rhosp-rhel-8.4-image-build-override                                                                      RHEL 8.4 Override
~~~
Comment 10 Brendan Shephard 2023-04-15 02:03:25 UTC 
Ack, let's close it off. Thanks for the confirmation.