Description of problem: OSP9 -> OSP10 -> OSP11 upgrade fails during major-upgrade-composable-steps because keystone admin is not reachable Version-Release number of selected component (if applicable): How reproducible: 4/4 Steps to Reproduce: 1. Deploy OSP9 2. Upgrade to latest OSP10 3. Upgrade to OSP11 Actual results: major-upgrade-composable-steps fails: Error: Failed to apply catalog: Execution of '/usr/bin/openstack domain list --quiet --format csv' returned 1: Unable to establish connection to http://controller-0.ctlplane.localdomain:35357/v3/domains?: HTTPConnectionPool(host='controller-0.ctlplane.localdomain', port=35357): Max retries exceeded with url: /v3/domains (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x3d96990>: Failed to establish a new connection: [Errno 111] Connection refused',)) (tried 46, for a total of 170 seconds) Expected results: Upgrade completes fine. Additional info: httpd is unable to start because it's missing mod_ssl so Keystone is unreachable. This seems to be a regression introduced in the 2017-04-24 build.
Reproduced.
[root@controller-0 ~]# httpd httpd: Syntax error on line 38 of /etc/httpd/conf/httpd.conf: Syntax error on line 1 of /etc/httpd/conf.modules.d/ssl.load: Cannot load modules/mod_ssl.so into server: /etc/httpd/modules/mod_ssl.so: cannot open shared object file: No such file or directory [root@controller-0 ~]# rpm -qa | grep mod_ssl [root@controller-0 ~]# cat /etc/httpd/conf.modules.d/ssl.load LoadModule ssl_module modules/mod_ssl.so [root@controller-0 ~]# [root@controller-0 ~]# systemctl status httpd ● httpd.service - The Apache HTTP Server Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled) Drop-In: /usr/lib/systemd/system/httpd.service.d └─openstack-dashboard.conf Active: failed (Result: exit-code) since Wed 2017-04-26 18:53:53 UTC; 16min ago Docs: man:httpd(8) man:apachectl(8) Process: 407849 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=1/FAILURE) Process: 407846 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, status=1/FAILURE) Process: 407791 ExecStartPre=/usr/bin/python /usr/share/openstack-dashboard/manage.py compress --force (code=exited, status=0/SUCCESS) Process: 407761 ExecStartPre=/usr/bin/python /usr/share/openstack-dashboard/manage.py collectstatic --noinput --clear (code=exited, status=0/SUCCESS) Main PID: 407846 (code=exited, status=1/FAILURE) Apr 26 18:53:45 controller-0.localdomain python[407761]: Copying '/usr/share/javascript/jquery_ui/themes/ui-lightness/images/animated-overlay.gif' Apr 26 18:53:45 controller-0.localdomain python[407761]: Copying '/usr/share/javascript/jquery_ui/themes/ui-lightness/images/ui-bg_gloss-wave_35_f6a828_500x100.png' Apr 26 18:53:45 controller-0.localdomain python[407761]: Copying '/usr/share/javascript/jquery_ui/themes/ui-lightness/images/ui-icons_ffd27a_256x240.png' Apr 26 18:53:45 controller-0.localdomain python[407761]: Copying '/usr/share/javascript/jquery_ui/themes/ui-lightness/images/ui-bg_glass_100_fdf5ce_1x400.png' Apr 26 18:53:53 controller-0.localdomain systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE Apr 26 18:53:53 controller-0.localdomain kill[407849]: kill: cannot find process "" Apr 26 18:53:53 controller-0.localdomain systemd[1]: httpd.service: control process exited, code=exited status=1 Apr 26 18:53:53 controller-0.localdomain systemd[1]: Failed to start The Apache HTTP Server. Apr 26 18:53:53 controller-0.localdomain systemd[1]: Unit httpd.service entered failed state. Apr 26 18:53:53 controller-0.localdomain systemd[1]: httpd.service failed.
Apr 26 18:56:56 controller-0 os-collect-config: #033[mNotice: /Stage[main]/Apache::Mod::Ssl/Apache::Mod[ssl]/Package[mod_ssl]/ensure: created#033[0m mod ssl package is missing and if you use tripleo::packages, the rpm provider is a noop so it never gets installed.
python-cradox and python-aodhclient are also missing and attempting to be installed via puppet but are being nooped by the use of OS::TripleO::Services::TripleoPackages
FYI, this was caused by https://review.openstack.org/#/c/458033/1 which added the inclusion of apache::mod::ssl to all the api classes in puppet-tripleo. This has the side effect of pulling (and activating) in mod_ssl which when combined with OS::TripleO::Services::TripleoPackages results in apache not being able to start because the mod_ssl module was not actually installed.
Adding a missing backport from master to stable/ocata.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:1475