Bug 1400971

Summary: keystone/httpd unable to start when using environments/updates/update-from-keystone-admin-internal-api.yaml
Product: Red Hat OpenStack Reporter: Cyril Lopez <cylopez>
Component: openstack-tripleo-heat-templatesAssignee: Jiri Stransky <jstransk>
Status: CLOSED CURRENTRELEASE QA Contact: Arik Chernetsky <achernet>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 10.0 (Newton)CC: aschultz, cylopez, emacchi, josorior, mburns, nkinder, rhel-osp-director-maint
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-11-21 19:43:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Cyril Lopez 2016-12-02 13:16:03 UTC 
Description of problem:

Deployement of 3 controller 3 compute. The deployment fail on the first controller cause httpd is not able to start cause :80 is used by haproxy.


Version-Release number of selected component (if applicable):
rhosp-director-images-10.0-20161130.1.el7ost.noarch
rhosp-director-images-ipa-10.0-20161130.1.el7ost.noarch
openstack-tripleo-heat-templates-5.1.0-7.el7ost.noarch

Log :

Warning: Scope(Haproxy::Config[haproxy]): haproxy: The $merge_options parameter will default to true in the next major release. Please review the documentation regarding the implications.
Error: Could not start Service[httpd]: Execution of '/usr/bin/systemctl start httpd' returned 1: Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.
Error: /Stage[main]/Apache::Service/Service[httpd]/ensure: change from stopped to running failed: Could not start Service[httpd]: Execution of '/usr/bin/systemctl start httpd' returned 1: Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journ
alctl -xe" for details.
Warning: /Stage[main]/Keystone::Deps/Anchor[keystone::service::end]: Skipping because of failed dependencies
Error: Could not prefetch keystone_tenant provider 'openstack': Execution of '/usr/bin/openstack project list --quiet --format csv --long' returned 1: Unable to establish connection to http://172.16.0.22:35357/v3/projects: HTTPConnectionPool(host='172.16.0.22', port=35357): Max retries exceeded with url: /v3/projec
ts (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x3817c10>: Failed to establish a new connection: [Errno 111] Connection refused',)) (tried 38, for a total of 170 seconds)
Warning: /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[service]: Skipping because of failed dependencies
Warning: /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[admin]: Skipping because of failed dependencies
Error: Could not prefetch keystone_role provider 'openstack': Execution of '/usr/bin/openstack role list --quiet --format csv' returned 1: Unable to establish connection to http://172.16.0.22:35357/v3/roles: HTTPConnectionPool(host='172.16.0.22', port=35357): Max retries exceeded with url: /v3/roles (Caused by NewC
onnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x300c050>: Failed to establish a new connection: [Errno 111] Connection refused',)) (tried 38, for a total of 170 seconds)
Warning: /Stage[main]/Keystone::Roles::Admin/Keystone_role[admin]: Skipping because of failed dependencies
Warning: /Stage[main]/Keystone::Roles::Admin/Keystone_user[admin]: Skipping because of failed dependencies
Warning: /Stage[main]/Keystone::Roles::Admin/Keystone_user_role[admin@admin]: Skipping because of failed dependencies
Error: Could not prefetch keystone_service provider 'openstack': Execution of '/usr/bin/openstack service list --quiet --format csv --long' returned 1: Unable to establish connection to http://172.16.0.22:35357/v3/services: HTTPConnectionPool(host='172.16.0.22', port=35357): Max retries exceeded with url: /v3/servi
ces (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x3bd9850>: Failed to establish a new connection: [Errno 111] Connection refused',)) (tried 38, for a total of 170 seconds)
Warning: /Stage[main]/Keystone::Endpoint/Keystone::Resource::Service_identity[keystone]/Keystone_service[keystone::identity]: Skipping because of failed dependencies
Error: Could not prefetch keystone_endpoint provider 'openstack': Execution of '/usr/bin/openstack endpoint list --quiet --format csv' returned 1: Unable to establish connection to http://172.16.0.22:35357/v3/endpoints: HTTPConnectionPool(host='172.16.0.22', port=35357): Max retries exceeded with url: /v3/endpoints
 (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x33228d0>: Failed to establish a new connection: [Errno 111] Connection refused',)) (tried 38, for a total of 170 seconds)



Manual Test :
[root@overcloud-controller-0 ~]# systemctl status httpd

* httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/httpd.service.d
           `-openstack-dashboard.conf
   Active: failed (Result: exit-code) since Fri 2016-12-02 12:23:20 GMT; 10s ago
     Docs: man:httpd(8)
           man:apachectl(8)
  Process: 201487 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=1/FAILURE)
  Process: 201484 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, status=1/FAILURE)
  Process: 200784 ExecStartPre=/usr/bin/python /usr/share/openstack-dashboard/manage.py compress --force (code=exited, status=0/SUCCESS)
  Process: 200448 ExecStartPre=/usr/bin/python /usr/share/openstack-dashboard/manage.py collectstatic --noinput --clear (code=exited, status=0/SUCCESS)
 Main PID: 201484 (code=exited, status=1/FAILURE)

Dec 02 12:23:05 overcloud-controller-0.localdomain python[200448]: Copying '/usr/share/javascript/jquery_ui/themes/south-street/images/ui-bg_inset-soft_15_2b2922_1x100.png'
Dec 02 12:23:05 overcloud-controller-0.localdomain python[200448]: Copying '/usr/share/javascript/jquery_ui/themes/south-street/images/ui-bg_highlight-soft_95_ffedad_1x100.png'
Dec 02 12:23:05 overcloud-controller-0.localdomain python[200448]: Copying '/usr/share/javascript/jquery_ui/themes/south-street/images/ui-bg_highlight-hard_100_f5f3e5_1x100.png'
Dec 02 12:23:05 overcloud-controller-0.localdomain python[200448]: Copying '/usr/share/javascript/jquery_ui/themes/south-street/images/ui-icons_847e71_256x240.png'
Dec 02 12:23:05 overcloud-controller-0.localdomain python[200448]: Copying '/usr/share/javascript/jquery_ui/themes/south-street/images/ui-bg_highlight-hard_95_cccccc_1x100.png'
Dec 02 12:23:20 overcloud-controller-0.localdomain systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE
Dec 02 12:23:20 overcloud-controller-0.localdomain systemd[1]: httpd.service: control process exited, code=exited status=1
Dec 02 12:23:20 overcloud-controller-0.localdomain systemd[1]: Failed to start The Apache HTTP Server.
Dec 02 12:23:20 overcloud-controller-0.localdomain systemd[1]: Unit httpd.service entered failed state.
Dec 02 12:23:20 overcloud-controller-0.localdomain systemd[1]: httpd.service failed.
[root@overcloud-controller-0 ~]# /usr/sbin/httpd $OPTIONS -DFOREGROUND
(98)Address already in use: AH00073: make_sock: unable to listen for connections on address [::]:80
(98)Address already in use: AH00073: make_sock: unable to listen for connections on address 0.0.0.0:80
no listening sockets available, shutting down
[root@overcloud-controller-0 ~]# systemctl stop haproxy
[root@overcloud-controller-0 ~]# systemctl start
[root@overcloud-controller-0 ~]# systemctl status httpd                 
* httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/httpd.service.d
           `-openstack-dashboard.conf
   Active: active (running) since Fri 2016-12-02 12:28:56 GMT; 50s ago
     Docs: man:httpd(8)
           man:apachectl(8)
 Main PID: 220283 (httpd)
   Status: "Total requests: 0; Current requests/sec: 0; Current traffic:   0 B/sec"
   CGroup: /system.slice/httpd.service
           |-220283 /usr/sbin/httpd -DFOREGROUND
           |-220285 keystone-admin  -DFOREGROUND
           |-220286 keystone-admin  -DFOREGROUND
           |-220287 keystone-admin  -DFOREGROUND
           |-220288 keystone-admin  -DFOREGROUND
           |-220289 keystone-main   -DFOREGROUND
           |-220290 keystone-main   -DFOREGROUND
           |-220291 keystone-main   -DFOREGROUND
           |-220292 keystone-main   -DFOREGROUND
           |-220293 /usr/sbin/httpd -DFOREGROUND
           |-220294 /usr/sbin/httpd -DFOREGROUND
           |-220295 /usr/sbin/httpd -DFOREGROUND
           |-220296 /usr/sbin/httpd -DFOREGROUND
           |-220297 /usr/sbin/httpd -DFOREGROUND
           |-220298 /usr/sbin/httpd -DFOREGROUND
           |-220299 /usr/sbin/httpd -DFOREGROUND
           `-220300 /usr/sbin/httpd -DFOREGROUND
Comment 2 Cyril Lopez 2016-12-02 14:18:45 UTC 
It's look the root cause :

[root@overcloud-controller-0 ~]# cat /etc/httpd/conf.d/15-default.conf 
# ************************************
# Vhost template in module puppetlabs-apache
# Managed by Puppet
# ************************************

<VirtualHost *:80>
  ServerName default
  ServerAdmin root@localhost

  ## Vhost docroot
  DocumentRoot "/var/www/html"

  ## Directories, there should at least be a declaration for /var/www/html

  <Directory "/var/www/html">
    Options Indexes FollowSymLinks MultiViews
    AllowOverride None
    Require all granted
  </Directory>

  ## Logging
  ErrorLog "/var/log/httpd/default_error.log"
  ServerSignature Off
  CustomLog "/var/log/httpd/access_log" combined 
  ## Script alias directives
  ScriptAlias /cgi-bin "/var/www/cgi-bin"
</VirtualHost>
Comment 3 Cyril Lopez 2016-12-02 14:37:40 UTC 
Using this env is the root cause :
/usr/share/openstack-tripleo-heat-templates/environments/updates/update-from-keystone-admin-internal-api.yaml

Cause ApacheNetwork: internal_api is not set and in /puppet/services/apache.yaml, 
apache::ip: {get_param: [ServiceNetMap, ApacheNetwork]} can be set.