Bug 1698555
| Summary: | overcloud node import hangs - mistral_executor is down due /var/lib/undercloud.conf missing | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Pavel Sedlák <psedlak> |
| Component: | openstack-tripleo-heat-templates | Assignee: | Michele Baldessari <michele> |
| Status: | CLOSED ERRATA | QA Contact: | Sasha Smolyak <ssmolyak> |
| Severity: | urgent | Docs Contact: | |
| Priority: | urgent | ||
| Version: | 15.0 (Stein) | CC: | aschultz, emacchi, jpichon, lhh, lvrabec, mburns, michele, yobshans, zcaplovi |
| Target Milestone: | beta | Keywords: | Triaged |
| Target Release: | 15.0 (Stein) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-tripleo-heat-templates-10.4.1-0.20190412000410.b934fdd.el8ost.noarch | Doc Type: | No Doc Update |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-09-21 11:21:11 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Pavel Sedlák
2019-04-10 15:25:26 UTC
I thought perhaps the undercloud.conf file is keeping the original home directory context permissions when mounted, when it should switch to a mistral or container-specific context. However, AlistairT ran ls -Z in an environment where this is working in enforcing mode and it appears like this should work even with the unconfined user_home_t context: podman exec -u root mistral_executor bash ls -lZ /var/lib/undercloud.conf -rwxr-xr-x. 1 1001 1001 unconfined_u:object_r:user_home_t:s0 891 Apr 4 10:54 /var/lib/undercloud.conf Investigating further. I doubt we want to give containers read access to the home directory in general. A) Enforcing on
# 68c1f09c2bfa is the mistral image
podman run -it --rm -user=root --net=host -e KOLLA_INSTALL_METATYPE=rhos -e KOLLA_INSTALL_TYPE=binary \
-e KOLLA_BASE_DISTRO=rhel -e KOLLA_CONFIG_STRATEGY=COPY_ALWAYS -e KOLLA_DISTRO_PYTHON_VERSION=3.6 \
-v /home/stack/undercloud.conf:/var/lib/undercloud.conf \
-v /var/lib/kolla/config_files/mistral_executor.json:/var/lib/kolla/config_files/config.json \
-v /var/lib/config-data/puppet-generated/mistral/:/var/lib/kolla/config_files/src 68c1f09c2bfa sh
[root@undercloud-0 ~]# sh x.sh
()[root@undercloud-0 /]$ kolla_set_configs
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
....snip....
INFO:__main__:Copying /var/lib/kolla/config_files/src/var/www/cgi-bin/mistral/app to /var/www/cgi-bin/mistral/app
ERROR:__main__:MissingRequiredSource: /var/lib/undercloud.conf file is not found
The error is a bit misleading because the file is actually there:
()[root@undercloud-0 /]$ ls -1 /var/lib/ |grep -i undercloud.conf
undercloud.conf
The problem is that we cannot access it:
()[root@undercloud-0 /]$ ls -lZ /var/lib/undercloud.conf
ls: cannot access '/var/lib/undercloud.conf': Permission denied
[root@undercloud-0 ~]# ls -ldZ /home/stack/ ; ls -lZ /home/stack/undercloud.conf
drwx------. 9 stack stack unconfined_u:object_r:user_home_dir_t:s0 4096 Apr 10 11:06 /home/stack/
-rwxr-xr-x. 1 stack stack unconfined_u:object_r:user_home_t:s0 891 Apr 10 10:23 /home/stack/undercloud.conf
The denied I see around this are all about dbus and sudo (which is likely https://bugs.launchpad.net/tripleo/+bug/1819461)
so not sure they are relevant (?):
type=AVC msg=audit(1554910473.290:10321): avc: denied { connectto } for pid=130205 comm="sudo" path="/run/dbus/system_bus_socket" scontext=system_u:system_r:container_t:s0:c363,c968 tcontext=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 tclass=unix_stream_socket permissive=0
type=AVC msg=audit(1554910473.293:10325): avc: denied { connectto } for pid=130205 comm="sudo" path="/run/dbus/system_bus_socket" scontext=system_u:system_r:container_t:s0:c363,c968 tcontext=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 tclass=unix_stream_socket permissive=0
B) Enforcing off
podman run -it --user=root --rm --net=host -e KOLLA_INSTALL_METATYPE=rhos -e KOLLA_INSTALL_TYPE=binary \
-e KOLLA_BASE_DISTRO=rhel -e KOLLA_CONFIG_STRATEGY=COPY_ALWAYS -e KOLLA_DISTRO_PYTHON_VERSION=3.6 \
-v /home/stack/undercloud.conf:/var/lib/undercloud.conf \
-v /var/lib/kolla/config_files/mistral_executor.json:/var/lib/kolla/config_files/config.json \
-v /var/lib/config-data/puppet-generated/mistral/:/var/lib/kolla/config_files/src 68c1f09c2bfa sh
()[root@undercloud-0 /]$ ls -lZ /var/lib/undercloud.conf
-rwxr-xr-x. 1 1001 1001 unconfined_u:object_r:user_home_t:s0 891 Apr 10 14:23 /var/lib/undercloud.conf
Seems to work correctly
What I do not fully understand is why we do not see any denials specific to these ls -l commands run as root user (without the sudo)?
Julie++'s suggestion seems to have worked:
1) Edited
/var/lib/tripleo-config/hashed-container-startup-config-step_4.json and added ',z' to the mistral_executor stanza for the undercloud.conf file only:
"/home/stack/undercloud.conf:/var/lib/undercloud.conf:ro,z",
2) Ran paunch again for step4
paunch --debug apply --default-runtime podman --file /var/lib/tripleo-config/hashed-container-startup-config-step_4.json --config-id tripleo_step4 --managed-by tripleo-Undercloud 2>&1 | tee /tmp/paunch.log
3) mistral_executor is up and running:
[root@undercloud-0 ~]# podman ps |grep mistral_ex
e220c54e027d 192.168.24.1:8787/rhosp15/openstack-mistral-executor:20190409.1 dumb-init --singl... About a minute ago Up About a minute ago mistral_executor
[root@undercloud-0 ~]# podman logs mistral_executor 2>&1|tail -n10
+ . kolla_extend_start
++ [[ ! -d /var/log/kolla/mistral ]]
++ mkdir -p /var/log/kolla/mistral
+++ stat -c %a /var/log/kolla/mistral
++ [[ 2755 != \7\5\5 ]]
++ chmod 755 /var/log/kolla/mistral
Running command: '/usr/bin/mistral-server --config-file=/etc/mistral/mistral.conf --log-file=/var/log/mistral/executor.log --server=executor'
++ . /usr/local/bin/kolla_mistral_extend_start
+ echo 'Running command: '\''/usr/bin/mistral-server --config-file=/etc/mistral/mistral.conf --log-file=/var/log/mistral/executor.log --server=executor'\'''
+ exec /usr/bin/mistral-server --config-file=/etc/mistral/mistral.conf --log-file=/var/log/mistral/executor.log --server=executor
*** Bug 1698540 has been marked as a duplicate of this bug. *** Node import passed successfully, no mistral in list of failed Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:2811 |