It seems that at least OSP 7 undercloud installation is broken due to RHEL 7.3. I couldn't confirm this in a lab yet, but the first case for this just roled in. I wonder if newer versions of OSP are affected as well. If this is the case, then the installation procedure will break for all customers who are not using e.g. satellite to lock their repos to a specific package version. https://bugs.launchpad.net/tripleo/+bug/1630740 Our documentation for OSP 7 - 9 states: ~~~ Red Hat Enterprise Linux 7.2 installed as the host operating system. ~~~ https://access.redhat.com/documentation/en/red-hat-openstack-platform/9/single/director-installation-and-usage/#sect-Undercloud_Requirements
I just installed a new box with RHEL 7.2, upgraded all packages to 7.3, installed OSP 9 with selinux enabled, and `openstack undercloud install` gives: ~~~ (...) + echo dib-run-parts Thu Nov 3 15:43:17 EDT 2016 Running /tmp/tmpk9v3v1/post-install.d/10-enable-init-scripts dib-run-parts Thu Nov 3 15:43:17 EDT 2016 Running /tmp/tmpk9v3v1/post-install.d/10-enable-init-scripts + target_tag=10-enable-init-scripts + date +%s.%N + /tmp/tmpk9v3v1/post-install.d/10-enable-init-scripts + target_tag=10-enable-init-scripts + date +%s.%N + output '10-enable-init-scripts completed' ++ date + echo dib-run-parts Thu Nov 3 15:43:17 EDT 2016 10-enable-init-scripts completed dib-run-parts Thu Nov 3 15:43:17 EDT 2016 10-enable-init-scripts completed + for target in '$targets' + output 'Running /tmp/tmpk9v3v1/post-install.d/86-selinux' ++ date + echo dib-run-parts Thu Nov 3 15:43:17 EDT 2016 Running /tmp/tmpk9v3v1/post-install.d/86-selinux dib-run-parts Thu Nov 3 15:43:17 EDT 2016 Running /tmp/tmpk9v3v1/post-install.d/86-selinux + target_tag=86-selinux + date +%s.%N + /tmp/tmpk9v3v1/post-install.d/86-selinux + set -o pipefail + '[' -x /usr/sbin/semanage ']' + mkdir -p /opt/stack/selinux-policy ++ dirname /tmp/tmpk9v3v1/post-install.d/86-selinux + checkmodule -M -m -o /tmp/ipxe.mod /tmp/tmpk9v3v1/post-install.d/../selinux/ipxe.te checkmodule: Module name ironic-ipxe is different than the output base filename ipxe checkmodule: loading policy configuration from /tmp/tmpk9v3v1/post-install.d/../selinux/ipxe.te INFO: 2016-11-03 15:43:18,003 -- ############### End stdout/stderr logging ############### ERROR: 2016-11-03 15:43:18,003 -- Hook FAILED. ERROR: 2016-11-03 15:43:18,003 -- Failed running command ['dib-run-parts', u'/tmp/tmpk9v3v1/post-install.d'] File "/usr/lib/python2.7/site-packages/instack/main.py", line 163, in main em.run() File "/usr/lib/python2.7/site-packages/instack/runner.py", line 79, in run self.run_hook(hook) File "/usr/lib/python2.7/site-packages/instack/runner.py", line 172, in run_hook raise Exception("Failed running command %s" % command) ERROR: 2016-11-03 15:43:18,003 -- None Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 844, in install _run_instack(instack_env) File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 728, in _run_instack _run_live_command(args, instack_env, 'instack') File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 406, in _run_live_command raise RuntimeError('%s failed. See log for details.' % name) RuntimeError: instack failed. See log for details. Command 'instack-install-undercloud' returned non-zero exit status 1 (...) ~~~
There are pending updates for Red Hat OpenStack Platform versions 7-9 which will resolve this. Until these are released, one workaround would be to downgrade checkpolicy to the 7.2-z version. If that is not possible, let me know.
So the problem is not with the fact that we do not have the fix .. the problem is that it's not in our repos: instack-undercloud-4.0.0-15.el7ost [stack@undercloud-4 ~]$ sudo yum update instack-undercloud Loaded plugins: priorities, search-disabled-repos No packages marked for update [stack@undercloud-4 ~]$ sudo yum update instack-undercloud Loaded plugins: priorities, search-disabled-repos No packages marked for update [stack@undercloud-4 ~]$ rpm -qa | grep instack-undercloud instack-undercloud-4.0.0-13.el7ost.noarch [stack@undercloud-4 ~]$ sudo yum repolist Loaded plugins: priorities, search-disabled-repos repo id repo name status rhel-7-server-extras-rpms/x86_64 Red Hat Enterprise Linux 7 Server - Extras (RPMs) 327 rhel-7-server-openstack-9-director-rpms/7Server/x86_64 Red Hat OpenStack Platform 9 director for RHEL 7 (RPMs) 44 rhel-7-server-openstack-9-rpms/7Server/x86_64 Red Hat OpenStack Platform 9 for RHEL 7 (RPMs) 665 rhel-7-server-optional-rpms/7Server/x86_64 Red Hat Enterprise Linux 7 Server - Optional (RPMs) 10,374 rhel-7-server-rh-common-rpms/7Server/x86_64 Red Hat Enterprise Linux 7 Server - RH Common (RPMs) 209 rhel-7-server-rpms/7Server/x86_64 Red Hat Enterprise Linux 7 Server (RPMs) 13,352 rhos-release RHOS Release 138 rhos-release-extras/7Server RHOS Release Extras 2 repolist: 25,111
I reiterate: the updates to OpenStack 7, 8, and 9 to resolve this have not been released yet. Thus, they will not appear in CDN repos.
CDN aka subscription manager
Hi, ~~~ Until these are released, one workaround would be to downgrade checkpolicy to the 7.2-z version. If that is not possible, let me know. ~~~ This leads into dependency hell. ~~~ m[root@undercloud-4 ~]# yum downgrade checkpolicy-2.1.12-6.el7.x86_64 Loaded plugins: priorities, search-disabled-repos Resolving Dependencies --> Running transaction check ---> Package checkpolicy.x86_64 0:2.1.12-6.el7 will be a downgrade ---> Package checkpolicy.x86_64 0:2.5-4.el7 will be erased --> Finished Dependency Resolution Error: Package: selinux-policy-devel-3.13.1-102.el7_3.4.noarch (@rhel-7-server-rpms) Requires: checkpolicy >= 2.5 Removing: checkpolicy-2.5-4.el7.x86_64 (@rhel-7-server-rpms) checkpolicy = 2.5-4.el7 Downgraded By: checkpolicy-2.1.12-6.el7.x86_64 (rhel-7-server-rpms) checkpolicy = 2.1.12-6.el7 ********************************************************************** yum can be configured to try to resolve such errors by temporarily enabling disabled repos and searching for missing dependencies. To enable this functionality please set 'notify_only=0' in /etc/yum/pluginconf.d/search-disabled-repos.conf ********************************************************************** Error: Package: selinux-policy-devel-3.13.1-102.el7_3.4.noarch (@rhel-7-server-rpms) Requires: checkpolicy >= 2.5 Removing: checkpolicy-2.5-4.el7.x86_64 (@rhel-7-server-rpms) checkpolicy = 2.5-4.el7 Downgraded By: checkpolicy-2.1.12-6.el7.x86_64 (rhel-7-server-rpms) checkpolicy = 2.1.12-6.el7 You could try using --skip-broken to work around the problem You could try running: rpm -Va --nofiles --nodigest ~~~ [root@undercloud-4 ~]# yum downgrade checkpolicy-2.1.12-6.el7.x86_64 selinux-policy-3.13.1-60.el7_2.9.noarch selinux-policy-devel-3.13.1-60.el7_2.9.noarch selinux-policy-doc-3.13.1-60.el7_2.9.noarch selinux-policy-minimum-3.13.1-60.el7_2.9.noarch selinux-policy-mls-3.13.1-60.el7_2.9.noarch selinux-policy-sandbox-3.13.1-60.el7_2.9.noarch selinux-policy-targeted-3.13.1-60.el7_2.9.noarch Loaded plugins: priorities, search-disabled-repos No Match for available package: selinux-policy-doc-3.13.1-60.el7_2.9.noarch No Match for available package: selinux-policy-minimum-3.13.1-60.el7_2.9.noarch No Match for available package: selinux-policy-mls-3.13.1-60.el7_2.9.noarch No Match for available package: selinux-policy-sandbox-3.13.1-60.el7_2.9.noarch Resolving Dependencies --> Running transaction check ---> Package checkpolicy.x86_64 0:2.1.12-6.el7 will be a downgrade ---> Package checkpolicy.x86_64 0:2.5-4.el7 will be erased ---> Package selinux-policy.noarch 0:3.13.1-60.el7_2.9 will be a downgrade ---> Package selinux-policy.noarch 0:3.13.1-102.el7_3.4 will be erased ---> Package selinux-policy-devel.noarch 0:3.13.1-60.el7_2.9 will be a downgrade ---> Package selinux-policy-devel.noarch 0:3.13.1-102.el7_3.4 will be erased ---> Package selinux-policy-targeted.noarch 0:3.13.1-60.el7_2.9 will be a downgrade ---> Package selinux-policy-targeted.noarch 0:3.13.1-102.el7_3.4 will be erased --> Processing Conflict: libsemanage-2.5-4.el7.x86_64 conflicts selinux-policy-base < 3.13.1-66 --> Processing Conflict: openssh-6.6.1p1-31.el7.x86_64 conflicts selinux-policy < 3.13.1-92 --> Finished Dependency Resolution Error: libsemanage conflicts with selinux-policy-targeted-3.13.1-60.el7_2.9.noarch Error: openssh conflicts with selinux-policy-3.13.1-60.el7_2.9.noarch You could try using --skip-broken to work around the problem You could try running: rpm -Va --nofiles --nodigest
The customer fixed it like this: ----------------------------------------------- 123 sudo vim ./usr/share/instack-undercloud/ipxe/selinux/ipxe.te 129 sudo vim ./usr/share/tripleo-image-elements/selinux/custom-policies/tripleo-selinux-mariadb.te 130 sudo vim ./usr/share/tripleo-image-elements/selinux/custom-policies/tripleo-selinux-rabbitmq.te I replaced the module name with the module name the undercloud install process complained about, for instance: [stack@director-osp7 /]$ diff ./usr/share/tripleo-image-elements/selinux/custom-policies/tripleo-selinux-mariadb.te ~/tripleo-selinux-mariadb.te 2c2 < module tripleo-selinux-mariadb 1.0; --- > module tripleo_selinux_mariadb 1.0; The package had the module name with _, while the module named using -. Hope this helps.
*** Bug 1391755 has been marked as a duplicate of this bug. ***
This issue affects RHEL 7.3 from Beta on with OSP versions 7-9. A change in RHEL 7.3 from beta on concerning the checkpolicy package causes it to error-out if the module name differs from the filename containing the module. The commit that was backported to RHEL 7.3 is here: https://github.com/SELinuxProject/selinux/commit/c6acfae4bc22586ad1dc259b0aad57fa6c5b43ef This breaks two packages: openstack-tripleo-image-elements instack-undercloud The patch for openstack-tripleo-image-elements is here: https://github.com/openstack/tripleo-image-elements/commit/3648c68a22b09bf0fc0604dba4f227ac3f3fbd75 The patch for instack-undercloud is here: https://github.com/openstack/instack-undercloud/commit/d0b205c489cb66b50146889e617cabc728dd8952
Added OSP7-9 bugzillas tracking this for visibility.
Are you sure the patches from comment #18 are enough? I got better results (on OSP8) by using this patch: --- /usr/share/tripleo-image-elements/selinux/custom-policies/tripleo-selinux-rabbitmq.te.orig 2016-02-26 16:15:15.000000000 -0500 +++ /usr/share/tripleo-image-elements/selinux/custom-policies/tripleo-selinux-rabbitmq.te 2016-11-03 22:35:56.955249595 -0400 @@ -1,5 +1,5 @@ -module tripleo_selinux_rabbitmq 1.0; +module tripleo-selinux-rabbitmq 1.0; require { type rabbitmq_var_lib_t; --- /usr/share/tripleo-image-elements/selinux/custom-policies/tripleo-selinux-mariadb.te.orig 2016-11-03 22:38:20.460422470 -0400 +++ /usr/share/tripleo-image-elements/selinux/custom-policies/tripleo-selinux-mariadb.te 2016-11-03 22:19:52.991136697 -0400 @@ -1,5 +1,5 @@ -module tripleo_selinux_mariadb 1.0; +module tripleo-selinux-mariadb 1.0; require { type haproxy_t; --- /usr/share/instack-undercloud/ipxe/post-install.d/86-selinux.orig 2016-11-03 22:17:05.076517456 -0400 +++ /usr/share/instack-undercloud/ipxe/post-install.d/86-selinux 2016-11-03 22:32:29.391683707 -0400 @@ -8,5 +8,5 @@ mkdir -p /opt/stack/selinux-policy # Compile the selinux policy -checkmodule -M -m -o /tmp/ipxe.mod $(dirname $0)/../selinux/ipxe.te -sudo semodule_package -m /tmp/ipxe.mod -o /opt/stack/selinux-policy/ipxe.pp +checkmodule -M -m -o /tmp/ironic-ipxe.mod $(dirname $0)/../selinux/ipxe.te +sudo semodule_package -m /tmp/ironic-ipxe.mod -o /opt/stack/selinux-policy/ipxe.pp
Note that my change to /usr/share/instack-undercloud/ipxe/post-install.d/86-selinux is different from the one in comment #18: https://github.com/openstack/instack-undercloud/commit/d0b205c489cb66b50146889e617cabc728dd8952 I actually had to use only -two- substitutions instead of three.
I can reproduce this on OSP9. The workaround specified in comment #21 works for me.
This affects RHOSP 7, 8, 9 and 10. RDO-Liberty, Mitaka, Newton and Ocata.
I tested with fresh RHEL7.3 install, for both OSP7 and OSP8. Comment #20 resolved the issue for me in RHOSP7 and RHOSP8. I first tried the patch details in comment #18, which did not work.
Created attachment 1218197 [details] actual fix included with ospd9
The actual fix included is more like comment #20. I'm not sure why the upstream one is different. James?
BTW, you can update instack-undercloud now and things should work.
Nevermind. I think it's because the patch in comment #20 did not include renaming ipxe.te -> ironic-ipxe.te. Upstream and downstream appear to match.
The patch in comment #18 should work, but you need both parts - rename of ipxe.te -> ironic-ipxe.te (which we'd need if we switched to more common SELinux policy building in the future) _and_ the change to 86-selinux. The reason the output file (ipxe.pp) was left alone is because other bits (os-apply-config, I think) rely on that module name. If we want to change it, we need to update instack-undercloud and os-apply-config at the same time. We can revisit that for OSP11; I wouldn't change that mid-stream.
Closing since all of the dependent bugzillas are closed.
It is still broken on CDN as of now.
(In reply to arkady kanevsky from comment #33) > It is still broken on CDN as of now. Can you elaborate on how this is broken? What versions of the following packages are installed" openstack-selinux instack-undercloud selinux-policy The issue here should be only visible when deploying RHEL 7.3 and content from before this week. It should not be visible with RHEL 7.2 and older bits or RHEL 7.2 and latest bits.
The versions we have now in our versionlock files are: openstack-selinux-0.7.3-3.el7ost.* instack-undercloud-4.0.0-13.el7ost.* selinux-policy-3.13.1-102.el7_3.4.* Initially we had selinux-policy as selinux-policy-3.13.1-60.el7_2.7.* . We updated it to see if it resolved, but we still run into the same issue.
Manisha can you add a bit of detail on what exactly is still breaking?
When yum update is called in our install-drector script we see it trying to rpm –q update firewalld even though the package was not originally installed. rpm –q –whatrequires firewalld returns “no package requires firewalld” --> Running transaction check ---> Package ipset-libs.x86_64 0:6.19-4.el7 will be installed --> Processing Conflict: firewalld-0.4.3.2-8.el7.noarch conflicts selinux-policy < 3.13.1-89 --> Finished Dependency Resolution Error: firewalld conflicts with selinux-policy-3.13.1-60.el7_2.7.noarch You could try using --skip-broken to work around the problem You could try running: rpm -Va --nofiles --nodigest Created symlink from /etc/systemd/system/basic.target.wants/iptables.service to /usr/lib/systemd/system/iptables.service. net.ipv4.ip_forward = 1 We resolved the issue by moving the package versions to those shown below and we’re able to get pass the initial issue. selinux-policy noarch 3.13.1-102.el7_3.4 rhel-7-server-rpms 412 k systemd x86_64 219-30.el7_3.3 rhel-7-server-rpms 5.2 M Updating for dependencies: libgudev1 x86_64 219-30.el7_3.3 rhel-7-server-rpms 76 k libselinux x86_64 2.5-6.el7 rhel-7-server-rpms 161 k libselinux-python x86_64 2.5-6.el7 rhel-7-server-rpms 234 k libselinux-ruby x86_64 2.5-6.el7 rhel-7-server-rpms 120 k libselinux-utils x86_64 2.5-6.el7 rhel-7-server-rpms 151 k libsemanage x86_64 2.5-4.el7 rhel-7-server-rpms 144 k libsemanage-python x86_64 2.5-4.el7 rhel-7-server-rpms 103 k libsepol x86_64 2.5-6.el7 rhel-7-server-rpms 288 k policycoreutils x86_64 2.5-8.el7 rhel-7-server-rpms 841 k policycoreutils-devel x86_64 2.5-8.el7 rhel-7-server-rpms 329 k policycoreutils-python x86_64 2.5-8.el7 rhel-7-server-rpms 444 k selinux-policy-targeted noarch 3.13.1-102.el7_3.4 rhel-7-server-rpms 6.4 M setools-libs x86_64 3.3.8-1.1.el7 rhel-7-server-rpms 610 k systemd-libs x86_64 219-30.el7_3.3 rhel-7-server-rpms 368 k systemd-sysv x86_64 219-30.el7_3.3 rhel-7-server-rpms 63 k After passing all the director node upgrade package issues, we hit the bug reported in comment #1 + for target in '$targets'^M + output 'Running /tmp/tmp5mdZzN/post-install.d/86-selinux'^M ++ date^M + echo dib-run-parts Tue Nov 8 14:14:10 CST 2016 Running /tmp/tmp5mdZzN/post-install.d/86-selinux^M dib-run-parts Tue Nov 8 14:14:10 CST 2016 Running /tmp/tmp5mdZzN/post-install.d/86-selinux^M + target_tag=86-selinux^M + date +%s.%N^M + /tmp/tmp5mdZzN/post-install.d/86-selinux^M + set -o pipefail^M + '[' -x /usr/sbin/semanage ']'^M + mkdir -p /opt/stack/selinux-policy^M ++ dirname /tmp/tmp5mdZzN/post-install.d/86-selinux^M + checkmodule -M -m -o /tmp/ipxe.mod /tmp/tmp5mdZzN/post-install.d/../selinux/ipxe.te^M checkmodule: Module name ironic-ipxe is different than the output base filename ipxe^M checkmodule: loading policy configuration from /tmp/tmp5mdZzN/post-install.d/../selinux/ipxe.te^M INFO: 2016-11-08 14:14:10,202 -- ############### End stdout/stderr logging ###############^M ERROR: 2016-11-08 14:14:10,203 -- Hook FAILED.^M ERROR: 2016-11-08 14:14:10,203 -- Failed running command ['dib-run-parts', u'/tmp/tmp5mdZzN/post-install.d']^M File "/usr/lib/python2.7/site-packages/instack/main.py", line 163, in main^M em.run()^M File "/usr/lib/python2.7/site-packages/instack/runner.py", line 79, in run^M self.run_hook(hook)^M File "/usr/lib/python2.7/site-packages/instack/runner.py", line 172, in run_hook^M raise Exception("Failed running command %s" % command)^M ERROR: 2016-11-08 14:14:10,203 -- None^M Traceback (most recent call last):^M File "<string>", line 1, in <module>^M File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 844, in install^M _run_instack(instack_env)^M File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 728, in _run_instack^M _run_live_command(args, instack_env, 'instack')^M File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 406, in _run_live_command^M raise RuntimeError('%s failed. See log for details.' % name)^M RuntimeError: instack failed. See log for details.^M
OK, the base problem is different, though you get to the same problem in the end. First, since you're version locking, keep the original selinux policy file. Otherwise, you're getting RHEL 7.3 selinux policy and will have to update more than just that one rpm. The problem that you're seeing is because firewalld is *not* version locked. I haven't tracked down why firewalld would get installed yet, though the failure to install it would mean that if it's an RPM requires, rpm -q --whatrequires wouldn't actually show it (because the package that requires it failed to install). You can try adding firewalld to your version lock with this entry: firewalld-0.3.9-14.el7.noarch.rpm That will lock you to the latest RHEL 7.2 firewalld and prevent the selinux issue from coming up. To summarize -- Change the selinux-policy version lock back to the original one (selinux-policy-3.13.1-60.el7_2.7) and add firewalld-0.3.9-14.el7.noarch to the version lock. That should resolve your issue. After doing that, can you re-run the rpm -q --whatrequires firewalld ? I'll keep looking from here as well, but you'll get there quicker than me, I expect. Thanks
So then we feel the issue is closed then and customers can upgrade to RHEL 7.3?
(In reply to Benjamin Schmaus from comment #40) > So then we feel the issue is closed then and customers can upgrade to RHEL > 7.3? For the general case with no version locking in place, yes, this is resolved. For people doing version locking, we're leaving this open until comment 38/39 are resolved
We initially tried adding firewalld to version lock with this entry: firewalld-0.3.9-14.el7.noarch.rpm However firewalld is NOT previously installed on our director node so the updates process tries to install the newer version: --> Running transaction check ---> Package firewalld-filesystem.noarch 0:0.4.3.2-8.el7 will be ---> installed Package ipset.x86_64 0:6.19-4.el7 will be installed ... When this occurred we try starting trying to satisfy the dependencies which led us to this issue. We will try again this morning.
(In reply to John Williams from comment #42) > We initially tried adding firewalld to version lock with this entry: > > firewalld-0.3.9-14.el7.noarch.rpm > > However firewalld is NOT previously installed on our director node so the > updates process tries to install the newer version: > > --> Running transaction check > ---> Package firewalld-filesystem.noarch 0:0.4.3.2-8.el7 will be > ---> installed Package ipset.x86_64 0:6.19-4.el7 will be installed > ... > > When this occurred we try starting trying to satisfy the dependencies which > led us to this issue. > > > We will try again this morning. We tried again this morning adding the firewalld to version lock and it worked. We are not sure why it didn't work when we tried earlier.
It appears all related issues to this are resolved now. Please reopen if this becomes a problem again.