Bug 1726483 - Overcloud deployment fails with container permissions error (ppc64le)
Summary: Overcloud deployment fails with container permissions error (ppc64le)
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: z12
: 13.0 (Queens)
Assignee: Tony Breeds
QA Contact: Jason Joyce
URL:
Whiteboard:
Depends On: 1709564 1723665 1752900
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-07-03 02:06 UTC by Tony Breeds
Modified: 2023-09-07 20:15 UTC (History)
26 users (show)

Fixed In Version: openstack-tripleo-common-8.7.1-5.el7ost openstack-tripleo-heat-templates-8.4.1-17.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1723665
Environment:
Last Closed: 2020-04-09 10:39:52 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 670241 0 'None' MERGED Increase the default memlock to 64MiB via ``DockerNovaComputeUlimit``. 2020-11-10 12:04:37 UTC
Red Hat Issue Tracker OSP-28242 0 None None None 2023-09-07 20:15:05 UTC

Comment 11 Kha Do 2019-08-29 19:20:35 UTC
When I attempted to verify this BZ, I saw that the heat template was updated with the new values, but the container still failed to start with a permission denied error:

Aug 29 18:27:41 overcloud-novacomputeppc64le-0 os-collect-config: "Error running ['docker', 'run', '--name', 'nova_cell_v2_discover_hosts', '--label', 'config_id=tripleo_step5', '--label', 'contai
ner_name=nova_cell_v2_discover_hosts', '--label', 'managed_by=paunch', '--label', 'config_data={\"start_order\": 0, \"image\": \"registry.access.redhat.com/rhosp13/openstack-nova-compute:13.0-98\"
, \"environment\": [\"TRIPLEO_DEPLOY_IDENTIFIER=1567097889\"], \"command\": \"/usr/bin/bootstrap_host_exec nova_compute su nova -s /bin/bash -c \\'/docker-config-scripts/nova_cell_v2_discover_host
s.py\\'\", \"user\": \"root\", \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/ca-trust
/source/anchors:/etc/pki/ca-trust/source/anchors:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bund
le.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\", \"/etc/puppet:/etc/puppet:ro\", \"/var/lib/c
onfig-data/nova_libvirt/etc/my.cnf.d/:/etc/my.cnf.d/:ro\", \"/var/lib/config-data/nova_libvirt/etc/nova/:/etc/nova/:ro\", \"/var/log/containers/nova:/var/log/nova\", \"/var/lib/docker-config-scrip
ts/:/docker-config-scripts/\"], \"net\": \"host\", \"detach\": false}', '--env=TRIPLEO_DEPLOY_IDENTIFIER=1567097889', '--net=host', '--user=root', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/e
tc/localtime:/etc/localtime:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro', '--volume=/
etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/
etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro', '--volume=/etc/puppet:/etc/puppet:ro', '--volume=/var/lib/config-data/nova_
libvirt/etc/my.cnf.d/:/etc/my.cnf.d/:ro', '--volume=/var/lib/config-data/nova_libvirt/etc/nova/:/etc/nova/:ro', '--volume=/var/log/containers/nova:/var/log/nova', '--volume=/var/lib/docker-config-
scripts/:/docker-config-scripts/', 'registry.access.redhat.com/rhosp13/openstack-nova-compute:13.0-98', '/usr/bin/bootstrap_host_exec', 'nova_compute', 'su', 'nova', '-s', '/bin/bash', '-c', \"'/d
ocker-config-scripts/nova_cell_v2_discover_hosts.py'\"]. [1]",
Aug 29 18:27:41 overcloud-novacomputeppc64le-0 os-collect-config: "",
Aug 29 18:27:41 overcloud-novacomputeppc64le-0 os-collect-config: "stdout: ",
Aug 29 18:27:41 overcloud-novacomputeppc64le-0 os-collect-config: "stderr: su: cannot open session: Permission denied"



When I looked at the system logs from the compute node, I found these errors in the log:

Aug 29 18:27:41 overcloud-novacomputeppc64le-0 su[73823]: (to nova) root on none
Aug 29 18:27:41 overcloud-novacomputeppc64le-0 su[73823]: pam_limits(su:session): Could not set limit for 'memlock': Operation not permitted
Aug 29 18:27:41 overcloud-novacomputeppc64le-0 su[73823]: pam_systemd(su:session): Failed to connect to system bus: No such file or directory
Aug 29 18:27:41 overcloud-novacomputeppc64le-0 su[73823]: pam_unix(su:session): session opened for user nova by (uid=0)
Aug 29 18:27:41 overcloud-novacomputeppc64le-0 dockerd-current[39862]: su: cannot open session: Permission denied

Comment 13 Lon Hohberger 2019-09-04 10:43:23 UTC
According to our records, this should be resolved by openstack-tripleo-heat-templates-8.3.1-79.el7ost.  This build is available now.

Comment 15 Rajesh Tailor 2020-02-27 12:42:56 UTC
Hi Miguel,

It looks like that backporting [1] won't work now, as the code snippet [2] where you are getting error is removed with [3] in queens upstream as well as OSP13 downstream.


[1] https://review.opendev.org/#/c/666497/1/deployment/nova/nova-compute-container-puppet.yaml@677
[2] https://review.opendev.org/#/c/682644/1/docker/services/nova-compute.yaml@283
[3] https://code.engineering.redhat.com/gerrit/#/c/183250/6/docker/services/nova-compute.yaml

FYI the latest patch related to nova_cell_v2_discover container available in openstack-tripleo-heat-templates 8.4.1-17
[4] http://pkgs.devel.redhat.com/cgit/rpms/openstack-tripleo-heat-templates/commit/?h=rhos-13.0-rhel-7&id=21e8282e1269cd79e9b4528c433c78103232e15a

So it looks like you might be using older version then 8.4.1-17.

Comment 18 Lon Hohberger 2020-03-11 10:35:26 UTC
According to our records, this should be resolved by openstack-tripleo-common-8.7.1-12.el7ost.  This build is available now.

Comment 19 Lon Hohberger 2020-03-11 10:35:32 UTC
According to our records, this should be resolved by openstack-tripleo-heat-templates-8.4.1-42.el7ost.  This build is available now.


Note You need to log in before you can comment on or make changes to this bug.