Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1300800

Summary: http fail to start on ipv6 environment
Product: Red Hat OpenStack Reporter: Asaf Hirshberg <ahirshbe>
Component: openstack-tripleo-heat-templatesAssignee: Hugh Brock <hbrock>
Status: CLOSED ERRATA QA Contact: Asaf Hirshberg <ahirshbe>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 7.0 (Kilo)CC: dmacpher, gfidente, kbasil, mburns, michele, ohochman, rhel-osp-director-maint, roxenham, ushkalim
Target Milestone: y3   
Target Release: 7.0 (Kilo)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-0.8.6-112.el7ost Doc Type: Bug Fix
Doc Text:
The Puppet manifest in the Heat template collection started the HTTP service through systemd. This caused Pacemaker to fail starting the service. This fix modifies the manifest to only configure but not start the HTTP service so that Pacemaker can assume control over HTTP. Pacemaker now starts the service successfully.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2016-02-18 16:51:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Asaf Hirshberg 2016-01-21 18:19:02 UTC 
Description of problem:
HTTPd is fail to start on ospd7.3 ipv6 deployment. 

Version-Release number of selected component (if applicable):
pcs status:
...

Failed Actions:
* httpd_start_0 on overcloud-controller-0 'not running' (7): call=224, status=complete, exitreason='none',
    last-rc-change='Thu Jan 21 12:47:03 2016', queued=0ms, exec=2015ms
* httpd_start_0 on overcloud-controller-2 'not running' (7): call=223, status=complete, exitreason='none',
    last-rc-change='Thu Jan 21 12:47:03 2016', queued=0ms, exec=2015ms
* httpd_start_0 on overcloud-controller-1 'not running' (7): call=224, status=complete, exitreason='none',
    last-rc-change='Thu Jan 21 12:47:03 2016', queued=0ms, exec=2017ms

...

[root@overcloud-controller-0 ~]# cat /var/log/httpd/error_log 
[Thu Jan 21 03:48:02.380619 2016] [core:notice] [pid 29931] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Thu Jan 21 03:48:02.397335 2016] [suexec:notice] [pid 29931] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Jan 21 03:48:02.406938 2016] [auth_digest:notice] [pid 29931] AH01757: generating secret for digest authentication ...
[Thu Jan 21 03:48:02.412333 2016] [mpm_prefork:notice] [pid 29931] AH00163: Apache/2.4.6 (Red Hat Enterprise Linux) mod_wsgi/3.4 Python/2.7.5 configured -- resuming normal operations
[Thu Jan 21 03:48:02.412383 2016] [core:notice] [pid 29931] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Thu Jan 21 03:58:15.227343 2016] [mpm_prefork:notice] [pid 29931] AH00170: caught SIGWINCH, shutting down gracefully
[Thu Jan 21 04:01:39.130484 2016] [core:notice] [pid 30344] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Thu Jan 21 04:01:39.131606 2016] [suexec:notice] [pid 30344] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Jan 21 04:01:39.141258 2016] [auth_digest:notice] [pid 30344] AH01757: generating secret for digest authentication ...
[Thu Jan 21 04:01:39.147064 2016] [mpm_prefork:notice] [pid 30344] AH00163: Apache/2.4.6 (Red Hat Enterprise Linux) mod_wsgi/3.4 Python/2.7.5 configured -- resuming normal operations
[Thu Jan 21 04:01:39.147111 2016] [core:notice] [pid 30344] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Thu Jan 21 06:13:23.433452 2016] [mpm_prefork:notice] [pid 30344] AH00170: caught SIGWINCH, shutting down gracefully
[Thu Jan 21 07:13:58.309993 2016] [core:notice] [pid 27480] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Thu Jan 21 07:13:58.311106 2016] [suexec:notice] [pid 27480] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Jan 21 07:13:58.320779 2016] [auth_digest:notice] [pid 27480] AH01757: generating secret for digest authentication ...
[Thu Jan 21 07:13:58.326438 2016] [mpm_prefork:notice] [pid 27480] AH00163: Apache/2.4.6 (Red Hat Enterprise Linux) mod_wsgi/3.4 Python/2.7.5 configured -- resuming normal operations
[Thu Jan 21 07:13:58.326481 2016] [core:notice] [pid 27480] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Thu Jan 21 08:10:24.019764 2016] [mpm_prefork:notice] [pid 27480] AH00170: caught SIGWINCH, shutting down gracefully
[Thu Jan 21 08:23:57.922836 2016] [core:notice] [pid 29035] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Thu Jan 21 08:23:57.923953 2016] [suexec:notice] [pid 29035] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Jan 21 08:23:57.933575 2016] [auth_digest:notice] [pid 29035] AH01757: generating secret for digest authentication ...
[Thu Jan 21 08:23:57.939769 2016] [mpm_prefork:notice] [pid 29035] AH00163: Apache/2.4.6 (Red Hat Enterprise Linux) mod_wsgi/3.4 Python/2.7.5 configured -- resuming normal operations
[Thu Jan 21 08:23:57.939814 2016] [core:notice] [pid 29035] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Thu Jan 21 08:51:40.465329 2016] [mpm_prefork:notice] [pid 29035] AH00170: caught SIGWINCH, shutting down gracefully
[Thu Jan 21 08:53:33.314032 2016] [core:notice] [pid 11402] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Thu Jan 21 08:53:33.315163 2016] [suexec:notice] [pid 11402] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Jan 21 08:53:33.325029 2016] [auth_digest:notice] [pid 11402] AH01757: generating secret for digest authentication ...
[Thu Jan 21 08:53:33.331088 2016] [mpm_prefork:notice] [pid 11402] AH00163: Apache/2.4.6 (Red Hat Enterprise Linux) mod_wsgi/3.4 Python/2.7.5 configured -- resuming normal operations
[Thu Jan 21 08:53:33.331133 2016] [core:notice] [pid 11402] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Thu Jan 21 08:57:48.079428 2016] [mpm_prefork:notice] [pid 11402] AH00170: caught SIGWINCH, shutting down gracefully
[root@overcloud-controller-0 ~]# 

[root@overcloud-controller-0 ~]# apachectl configtest
Syntax OK


Additional info:
rhos-release-1.0.25-1.noarch.rpm
httpd-2.4.6-40.el7.x86_64

deployment: IPv6 BM , 3 controllers, 3 computes
Comment 2 Michele Baldessari 2016-01-22 14:29:15 UTC 
Just for completeness this is likely related to https://bugzilla.redhat.com/show_bug.cgi?id=1293942 (and it's 7.x clone):

    # more /usr/lib/systemd/system/httpd.service.d/openstack-dashboard.conf 
    [Service]
    ExecStartPre=/usr/bin/python /usr/share/openstack-dashboard/manage.py collectstatic --noinput --clear
    ExecStartPre=/usr/bin/python /usr/share/openstack-dashboard/manage.py compress --force
    ....

    - manage.py has this:
    import os
    import sys

    from django.core.management import execute_from_command_line  # noqa

    if __name__ == "__main__":
        os.environ.setdefault("DJANGO_SETTINGS_MODULE",
                              "openstack_dashboard.settings")

        execute_from_command_line(sys.argv)


    Which when run by hand gives:
    Traceback (most recent call last):
      File "/usr/share/openstack-dashboard/manage.py", line 23, in <module>
        execute_from_command_line(sys.argv) File "/usr/lib/python2.7/site-packages/django/core/management/__init__.py", line 338, in execute_from_command_line
        utility.execute()
      File "/usr/lib/python2.7/site-packages/django/core/management/__init__.py", line 312, in execute
        django.setup()
      File "/usr/lib/python2.7/site-packages/django/__init__.py", line 17, in setup 
        configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
      File "/usr/lib/python2.7/site-packages/django/utils/log.py", line 86, in configure_logging
        logging_config_func(logging_settings)
      File "/usr/lib64/python2.7/logging/config.py", line 803, in dictConfig
        dictConfigClass(config).configure()
      File "/usr/lib64/python2.7/logging/config.py", line 585, in configure
        '%r: %s' % (name, e))
    ValueError: Unable to configure handler 'syslog': [Errno 111] Connection refused
Comment 4 Asaf Hirshberg 2016-01-26 15:34:12 UTC 
Verified.

[root@puma33 ~]# rpm -qa|grep openstack-tripleo-heat
openstack-tripleo-heat-templates-0.8.6-112.el7ost.noarch

[root@overcloud-controller-1 ~]# systemctl status httpd
● httpd.service - Cluster Controlled httpd
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled)
  Drop-In: /run/systemd/system/httpd.service.d
           └─50-pacemaker.conf
        /usr/lib/systemd/system/httpd.service.d
           └─openstack-dashboard.conf
   Active: active (running) since Tue 2016-01-26 09:41:16 EST; 52min ago
     Docs: man:httpd(8)
           man:apachectl(8)
 Main PID: 5795 (httpd)
   Status: "Total requests: 23; Current requests/sec: 0; Current traffic:   0 
...
Comment 6 Rhys Oxenham 2016-02-01 14:14:10 UTC 
Problem also exists in Packstack environments.

Issuing a 'restorecon /etc/machine-id; systemctl restart systemd-journald; systemctl restart httpd' allowed me to proceed here...

[root@rhelosp7 ~(keystone_admin)]# ls -laZ /etc/machine-id
-rw-r--r--. root root system_u:object_r:machineid_t:s0 /etc/machine-id

[root@rhelosp7 ~(keystone_admin)]# systemctl status httpd
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/httpd.service.d
           └─openstack-dashboard.conf
   Active: active (running) since Mon 2016-02-01 14:12:55 GMT; 2s ago
     Docs: man:httpd(8)
           man:apachectl(8)
  Process: 5544 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=1/FAILURE)
  Process: 5606 ExecStartPre=/usr/bin/python /usr/share/openstack-dashboard/manage.py compress --force (code=exited, status=0/SUCCESS)
  Process: 5598 ExecStartPre=/usr/bin/python /usr/share/openstack-dashboard/manage.py collectstatic --noinput (code=exited, status=0/SUCCESS)
 Main PID: 5621 (httpd)
   Status: "Processing requests..."
   CGroup: /system.slice/httpd.service
           ├─5621 /usr/sbin/httpd -DFOREGROUND
           ├─5624 keystone-admin  -DFOREGROUND
           ├─5625 keystone-main   -DFOREGROUND
           ├─5626 /usr/sbin/httpd -DFOREGROUND
           ├─5627 /usr/sbin/httpd -DFOREGROUND
           ├─5628 /usr/sbin/httpd -DFOREGROUND
           ├─5629 /usr/sbin/httpd -DFOREGROUND
           ├─5630 /usr/sbin/httpd -DFOREGROUND
           ├─5631 /usr/sbin/httpd -DFOREGROUND
           ├─5632 /usr/sbin/httpd -DFOREGROUND
           ├─5633 /usr/sbin/httpd -DFOREGROUND
           ├─5634 /usr/sbin/httpd -DFOREGROUND
           ├─5635 /usr/sbin/httpd -DFOREGROUND
           └─5636 /usr/sbin/httpd -DFOREGROUND
Comment 8 errata-xmlrpc 2016-02-18 16:51:25 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0264.html