Bug 1300800 - http fail to start on ipv6 environment
Summary: http fail to start on ipv6 environment
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 7.0 (Kilo)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: y3
: 7.0 (Kilo)
Assignee: Hugh Brock
QA Contact: Asaf Hirshberg
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-01-21 18:19 UTC by Asaf Hirshberg
Modified: 2016-02-18 16:51 UTC (History)
9 users (show)

Fixed In Version: openstack-tripleo-heat-templates-0.8.6-112.el7ost
Doc Type: Bug Fix
Doc Text:
The Puppet manifest in the Heat template collection started the HTTP service through systemd. This caused Pacemaker to fail starting the service. This fix modifies the manifest to only configure but not start the HTTP service so that Pacemaker can assume control over HTTP. Pacemaker now starts the service successfully.
Clone Of:
Environment:
Last Closed: 2016-02-18 16:51:25 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 206463 0 None None None 2016-01-22 15:21:26 UTC
Red Hat Product Errata RHBA-2016:0264 0 normal SHIPPED_LIVE Red Hat Enterprise Linux OSP 7 director Bug Fix Advisory 2016-02-18 21:41:29 UTC

Description Asaf Hirshberg 2016-01-21 18:19:02 UTC
Description of problem:
HTTPd is fail to start on ospd7.3 ipv6 deployment. 

Version-Release number of selected component (if applicable):
pcs status:
...

Failed Actions:
* httpd_start_0 on overcloud-controller-0 'not running' (7): call=224, status=complete, exitreason='none',
    last-rc-change='Thu Jan 21 12:47:03 2016', queued=0ms, exec=2015ms
* httpd_start_0 on overcloud-controller-2 'not running' (7): call=223, status=complete, exitreason='none',
    last-rc-change='Thu Jan 21 12:47:03 2016', queued=0ms, exec=2015ms
* httpd_start_0 on overcloud-controller-1 'not running' (7): call=224, status=complete, exitreason='none',
    last-rc-change='Thu Jan 21 12:47:03 2016', queued=0ms, exec=2017ms

...

[root@overcloud-controller-0 ~]# cat /var/log/httpd/error_log 
[Thu Jan 21 03:48:02.380619 2016] [core:notice] [pid 29931] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Thu Jan 21 03:48:02.397335 2016] [suexec:notice] [pid 29931] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Jan 21 03:48:02.406938 2016] [auth_digest:notice] [pid 29931] AH01757: generating secret for digest authentication ...
[Thu Jan 21 03:48:02.412333 2016] [mpm_prefork:notice] [pid 29931] AH00163: Apache/2.4.6 (Red Hat Enterprise Linux) mod_wsgi/3.4 Python/2.7.5 configured -- resuming normal operations
[Thu Jan 21 03:48:02.412383 2016] [core:notice] [pid 29931] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Thu Jan 21 03:58:15.227343 2016] [mpm_prefork:notice] [pid 29931] AH00170: caught SIGWINCH, shutting down gracefully
[Thu Jan 21 04:01:39.130484 2016] [core:notice] [pid 30344] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Thu Jan 21 04:01:39.131606 2016] [suexec:notice] [pid 30344] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Jan 21 04:01:39.141258 2016] [auth_digest:notice] [pid 30344] AH01757: generating secret for digest authentication ...
[Thu Jan 21 04:01:39.147064 2016] [mpm_prefork:notice] [pid 30344] AH00163: Apache/2.4.6 (Red Hat Enterprise Linux) mod_wsgi/3.4 Python/2.7.5 configured -- resuming normal operations
[Thu Jan 21 04:01:39.147111 2016] [core:notice] [pid 30344] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Thu Jan 21 06:13:23.433452 2016] [mpm_prefork:notice] [pid 30344] AH00170: caught SIGWINCH, shutting down gracefully
[Thu Jan 21 07:13:58.309993 2016] [core:notice] [pid 27480] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Thu Jan 21 07:13:58.311106 2016] [suexec:notice] [pid 27480] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Jan 21 07:13:58.320779 2016] [auth_digest:notice] [pid 27480] AH01757: generating secret for digest authentication ...
[Thu Jan 21 07:13:58.326438 2016] [mpm_prefork:notice] [pid 27480] AH00163: Apache/2.4.6 (Red Hat Enterprise Linux) mod_wsgi/3.4 Python/2.7.5 configured -- resuming normal operations
[Thu Jan 21 07:13:58.326481 2016] [core:notice] [pid 27480] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Thu Jan 21 08:10:24.019764 2016] [mpm_prefork:notice] [pid 27480] AH00170: caught SIGWINCH, shutting down gracefully
[Thu Jan 21 08:23:57.922836 2016] [core:notice] [pid 29035] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Thu Jan 21 08:23:57.923953 2016] [suexec:notice] [pid 29035] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Jan 21 08:23:57.933575 2016] [auth_digest:notice] [pid 29035] AH01757: generating secret for digest authentication ...
[Thu Jan 21 08:23:57.939769 2016] [mpm_prefork:notice] [pid 29035] AH00163: Apache/2.4.6 (Red Hat Enterprise Linux) mod_wsgi/3.4 Python/2.7.5 configured -- resuming normal operations
[Thu Jan 21 08:23:57.939814 2016] [core:notice] [pid 29035] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Thu Jan 21 08:51:40.465329 2016] [mpm_prefork:notice] [pid 29035] AH00170: caught SIGWINCH, shutting down gracefully
[Thu Jan 21 08:53:33.314032 2016] [core:notice] [pid 11402] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Thu Jan 21 08:53:33.315163 2016] [suexec:notice] [pid 11402] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Jan 21 08:53:33.325029 2016] [auth_digest:notice] [pid 11402] AH01757: generating secret for digest authentication ...
[Thu Jan 21 08:53:33.331088 2016] [mpm_prefork:notice] [pid 11402] AH00163: Apache/2.4.6 (Red Hat Enterprise Linux) mod_wsgi/3.4 Python/2.7.5 configured -- resuming normal operations
[Thu Jan 21 08:53:33.331133 2016] [core:notice] [pid 11402] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Thu Jan 21 08:57:48.079428 2016] [mpm_prefork:notice] [pid 11402] AH00170: caught SIGWINCH, shutting down gracefully
[root@overcloud-controller-0 ~]# 

[root@overcloud-controller-0 ~]# apachectl configtest
Syntax OK


Additional info:
rhos-release-1.0.25-1.noarch.rpm
httpd-2.4.6-40.el7.x86_64

deployment: IPv6 BM , 3 controllers, 3 computes

Comment 2 Michele Baldessari 2016-01-22 14:29:15 UTC
Just for completeness this is likely related to https://bugzilla.redhat.com/show_bug.cgi?id=1293942 (and it's 7.x clone):

    # more /usr/lib/systemd/system/httpd.service.d/openstack-dashboard.conf 
    [Service]
    ExecStartPre=/usr/bin/python /usr/share/openstack-dashboard/manage.py collectstatic --noinput --clear
    ExecStartPre=/usr/bin/python /usr/share/openstack-dashboard/manage.py compress --force
    ....

    - manage.py has this:
    import os
    import sys

    from django.core.management import execute_from_command_line  # noqa

    if __name__ == "__main__":
        os.environ.setdefault("DJANGO_SETTINGS_MODULE",
                              "openstack_dashboard.settings")

        execute_from_command_line(sys.argv)


    Which when run by hand gives:
    Traceback (most recent call last):
      File "/usr/share/openstack-dashboard/manage.py", line 23, in <module>
        execute_from_command_line(sys.argv) File "/usr/lib/python2.7/site-packages/django/core/management/__init__.py", line 338, in execute_from_command_line
        utility.execute()
      File "/usr/lib/python2.7/site-packages/django/core/management/__init__.py", line 312, in execute
        django.setup()
      File "/usr/lib/python2.7/site-packages/django/__init__.py", line 17, in setup 
        configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
      File "/usr/lib/python2.7/site-packages/django/utils/log.py", line 86, in configure_logging
        logging_config_func(logging_settings)
      File "/usr/lib64/python2.7/logging/config.py", line 803, in dictConfig
        dictConfigClass(config).configure()
      File "/usr/lib64/python2.7/logging/config.py", line 585, in configure
        '%r: %s' % (name, e))
    ValueError: Unable to configure handler 'syslog': [Errno 111] Connection refused

Comment 4 Asaf Hirshberg 2016-01-26 15:34:12 UTC
Verified.

[root@puma33 ~]# rpm -qa|grep openstack-tripleo-heat
openstack-tripleo-heat-templates-0.8.6-112.el7ost.noarch

[root@overcloud-controller-1 ~]# systemctl status httpd
● httpd.service - Cluster Controlled httpd
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled)
  Drop-In: /run/systemd/system/httpd.service.d
           └─50-pacemaker.conf
        /usr/lib/systemd/system/httpd.service.d
           └─openstack-dashboard.conf
   Active: active (running) since Tue 2016-01-26 09:41:16 EST; 52min ago
     Docs: man:httpd(8)
           man:apachectl(8)
 Main PID: 5795 (httpd)
   Status: "Total requests: 23; Current requests/sec: 0; Current traffic:   0 
...

Comment 6 Rhys Oxenham 2016-02-01 14:14:10 UTC
Problem also exists in Packstack environments.

Issuing a 'restorecon /etc/machine-id; systemctl restart systemd-journald; systemctl restart httpd' allowed me to proceed here...

[root@rhelosp7 ~(keystone_admin)]# ls -laZ /etc/machine-id
-rw-r--r--. root root system_u:object_r:machineid_t:s0 /etc/machine-id

[root@rhelosp7 ~(keystone_admin)]# systemctl status httpd
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/httpd.service.d
           └─openstack-dashboard.conf
   Active: active (running) since Mon 2016-02-01 14:12:55 GMT; 2s ago
     Docs: man:httpd(8)
           man:apachectl(8)
  Process: 5544 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=1/FAILURE)
  Process: 5606 ExecStartPre=/usr/bin/python /usr/share/openstack-dashboard/manage.py compress --force (code=exited, status=0/SUCCESS)
  Process: 5598 ExecStartPre=/usr/bin/python /usr/share/openstack-dashboard/manage.py collectstatic --noinput (code=exited, status=0/SUCCESS)
 Main PID: 5621 (httpd)
   Status: "Processing requests..."
   CGroup: /system.slice/httpd.service
           ├─5621 /usr/sbin/httpd -DFOREGROUND
           ├─5624 keystone-admin  -DFOREGROUND
           ├─5625 keystone-main   -DFOREGROUND
           ├─5626 /usr/sbin/httpd -DFOREGROUND
           ├─5627 /usr/sbin/httpd -DFOREGROUND
           ├─5628 /usr/sbin/httpd -DFOREGROUND
           ├─5629 /usr/sbin/httpd -DFOREGROUND
           ├─5630 /usr/sbin/httpd -DFOREGROUND
           ├─5631 /usr/sbin/httpd -DFOREGROUND
           ├─5632 /usr/sbin/httpd -DFOREGROUND
           ├─5633 /usr/sbin/httpd -DFOREGROUND
           ├─5634 /usr/sbin/httpd -DFOREGROUND
           ├─5635 /usr/sbin/httpd -DFOREGROUND
           └─5636 /usr/sbin/httpd -DFOREGROUND

Comment 8 errata-xmlrpc 2016-02-18 16:51:25 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0264.html


Note You need to log in before you can comment on or make changes to this bug.