Bug 1207098 - [RDO] packstack installation failed with "Error: /Stage[main]/Apache::Service/Service[httpd]: Failed to call refresh: Could not start Service[httpd]: Execution of '/sbin/service httpd start' returned 1: Redirecting to /bin/systemctl start httpd.service"
Summary: [RDO] packstack installation failed with "Error: /Stage[main]/Apache::Service...
Keywords:
Status: CLOSED EOL
Alias: None
Product: RDO
Classification: Community
Component: openstack-packstack
Version: Juno
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
: Kilo
Assignee: Martin Magr
QA Contact: Shai Revivo
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-03-30 08:49 UTC by Udayendu Kar
Modified: 2018-06-05 07:34 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-05-19 15:50:38 UTC
dschoenb: needinfo-


Attachments (Terms of Use)
Fresh setup failed logs (7.59 KB, text/plain)
2015-04-02 07:06 UTC, Udayendu Kar
no flags Details
audit.log after setup failure (1.04 MB, text/plain)
2015-04-02 07:07 UTC, Udayendu Kar
no flags Details
httpd failure logs (26.75 KB, application/octet-stream)
2015-04-02 07:09 UTC, Udayendu Kar
no flags Details
Latest audit logs on Fedora 20 (3.81 MB, text/plain)
2015-04-22 10:48 UTC, Udayendu Kar
no flags Details

Description Udayendu Kar 2015-03-30 08:49:34 UTC
Description of problem:
packstack installation failed with the below error message:

====
2015-03-30 13:02:52::ERROR::run_setup::1018::root:: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/packstack/installer/run_setup.py", line 993, in main
    single_step_aio_install(options, logFile)
  File "/usr/lib/python2.7/site-packages/packstack/installer/run_setup.py", line 761, in single_step_aio_install
    single_step_install(options, logFile)
  File "/usr/lib/python2.7/site-packages/packstack/installer/run_setup.py", line 792, in single_step_install
    _main(options, answerfilepath, logFile)
  File "/usr/lib/python2.7/site-packages/packstack/installer/run_setup.py", line 655, in _main
    runSequences()
  File "/usr/lib/python2.7/site-packages/packstack/installer/run_setup.py", line 626, in runSequences
    controller.runAllSequences()
  File "/usr/lib/python2.7/site-packages/packstack/installer/setup_controller.py", line 81, in runAllSequences
    sequence.run(config=self.CONF, messages=self.MESSAGES)
  File "/usr/lib/python2.7/site-packages/packstack/installer/core/sequences.py", line 109, in run
    step.run(config=config, messages=messages)
  File "/usr/lib/python2.7/site-packages/packstack/installer/core/sequences.py", line 50, in run
    self.function(config, messages)
  File "/usr/lib/python2.7/site-packages/packstack/plugins/puppet_950.py", line 254, in apply_puppet_manifest
    wait_for_puppet(currently_running, messages)
  File "/usr/lib/python2.7/site-packages/packstack/plugins/puppet_950.py", line 126, in wait_for_puppet
    validate_logfile(log)
  File "/usr/lib/python2.7/site-packages/packstack/modules/puppet.py", line 107, in validate_logfile
    raise PuppetError(message)
PuppetError: Error appeared during Puppet run: xxx.xxx.xxx.xxx_keystone.pp
Error: /Stage[main]/Apache::Service/Service[httpd]: Failed to call refresh: Could not start Service[httpd]: Execution of '/sbin/service httpd start' returned 1: Redirecting to /bin/systemctl start  httpd.service
You will find full trace in log /var/tmp/packstack/20150330-121325-U4piRK/manifests/xxx.xxx.xxx.xxx_keystone.pp.log
====

Version-Release number of selected component (if applicable):
Fedora Server 21
RDO OpenStack Juno

How reproducible:
100%

Steps to Reproduce:
1. Install a Fedora 21 server edition OS
2. Register the RDO Juno openstack repo and install the steps as mentioned in https://www.rdoproject.org/Quickstart . While doing "packstack --allinone", it will fail with the below error message:

==--==
PuppetError: Error appeared during Puppet run: xxx.xxx.xxx.xxx_keystone.pp
Error: /Stage[main]/Apache::Service/Service[httpd]: Failed to call refresh: Could not start Service[httpd]: Execution of '/sbin/service httpd start' returned 1: Redirecting to /bin/systemctl start  httpd.service
You will find full trace in log /var/tmp/packstack/20150330-121325-U4piRK/manifests/xxx.xxx.xxx.xxx_keystone.pp.log
==--==

Actual results:
"packstack --allinone" failed as it failed to start the httpd service.

Expected results:
"packstack --allinone" should work well.

Additional info:

During the analysis observed that selinux is creating issue while trying to bind with a specific port:

==--==
# service httpd status
Redirecting to /bin/systemctl status  httpd.service
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled)
   Active: failed (Result: exit-code) since Mon 2015-03-30 13:50:29 IST; 33s ago
  Process: 893 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, status=1/FAILURE)
 Main PID: 893 (code=exited, status=1/FAILURE)

Mar 30 13:50:29 localhost.localdomain httpd[893]: (13)Permission denied: AH00072: make_sock: could not bind to address [::]:5000
Mar 30 13:50:29 localhost.localdomain httpd[893]: (13)Permission denied: AH00072: make_sock: could not bind to address 0.0.0.0:5000
Mar 30 13:50:29 localhost.localdomain httpd[893]: no listening sockets available, shutting down
Mar 30 13:50:29 localhost.localdomain httpd[893]: AH00015: Unable to open logs
Mar 30 13:50:29 localhost.localdomain systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE
Mar 30 13:50:29 localhost.localdomain systemd[1]: Failed to start The Apache HTTP Server.
Mar 30 13:50:29 localhost.localdomain systemd[1]: Unit httpd.service entered failed state.
Mar 30 13:50:29 localhost.localdomain systemd[1]: httpd.service failed.
==--==

To fix this issue I just set the selinux into permissive mode and everything started working:

Steps to fix:
------------
# getenforce 
Enforcing

# setenforce 0

# getenforce 
Permissive

# /bin/systemctl start  httpd.service

# /bin/systemctl status  httpd.service
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled)
   Active: active (running) since Mon 2015-03-30 13:52:00 IST; 12s ago
 Main PID: 1599 (httpd)
   Status: "Total requests: 0; Idle/Busy workers 100/0;Requests/sec: 0; Bytes served/sec:   0 B/sec"
   CGroup: /system.slice/httpd.service
           ├─1599 /usr/sbin/httpd -DFOREGROUND
           ├─1600 /usr/sbin/httpd -DFOREGROUND
           ├─1601 /usr/sbin/httpd -DFOREGROUND
           ├─1602 /usr/sbin/httpd -DFOREGROUND
           ├─1603 /usr/sbin/httpd -DFOREGROUND
           ├─1604 /usr/sbin/httpd -DFOREGROUND
           ├─1605 /usr/sbin/httpd -DFOREGROUND
           ├─1606 /usr/sbin/httpd -DFOREGROUND
           ├─1607 /usr/sbin/httpd -DFOREGROUND
           ├─1608 /usr/sbin/httpd -DFOREGROUND
           └─1609 /usr/sbin/httpd -DFOREGROUND

Then set "SELINUX=permissive" in "/etc/sysconfig/selinux" file and rebooted the server though its not a good way in production.

Comment 1 Udayendu Kar 2015-03-30 08:50:34 UTC
This bug looks similar to BZ#1174749

Comment 2 Lars Kellogg-Stedman 2015-03-30 19:24:54 UTC
Would you attach your /var/log/audit/audit.log containing the selinux errors to this bug?  Thanks!

Comment 3 Udayendu Kar 2015-03-31 14:00:06 UTC
Lars,

I destroyed that setup but this issue is reproducible. So I will reproduce it and will attach the required logs for your analysis.

Thanks,
Uday

Comment 4 Udayendu Kar 2015-04-02 07:06:57 UTC
Created attachment 1009995 [details]
Fresh setup failed logs

Comment 5 Udayendu Kar 2015-04-02 07:07:46 UTC
Created attachment 1009996 [details]
audit.log after setup failure

Comment 6 Udayendu Kar 2015-04-02 07:09:20 UTC
Created attachment 1009997 [details]
httpd failure logs

Comment 7 Udayendu Kar 2015-04-02 07:10:03 UTC
Hi Lars,

Please find all the required attached logs.

Thanks,
Uday

Comment 8 Rich Megginson 2015-04-06 22:31:11 UTC
https://bugzilla.redhat.com/show_bug.cgi?id=1138424

What version of selinux-policy are you using?  rpm -qa|grep selinux

Comment 9 Rich Megginson 2015-04-06 23:05:04 UTC
There are no httpd related avcs in audit.log

Comment 10 Udayendu Kar 2015-04-07 07:33:03 UTC
# rpm -qa | grep -i selinux
rpm-plugin-selinux-4.12.0.1-5.fc21.x86_64
selinux-policy-3.13.1-105.9.fc21.noarch
libselinux-ruby-2.3-5.fc21.x86_64
libselinux-python-2.3-5.fc21.x86_64
selinux-policy-targeted-3.13.1-105.9.fc21.noarch
libselinux-utils-2.3-5.fc21.x86_64
libselinux-2.3-5.fc21.x86_64


# cat /etc/redhat-release 
Fedora release 21 (Twenty One)

# uname -a
Linux localhost.localdomain 3.19.3-200.fc21.x86_64 #1 SMP Thu Mar 26 21:39:42 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

I also didnt get any clue in the audit logs and thats the reason opened this bz. This issue is 100% reproducible. So if you want you can try to reproduce it.

Let me know if you need any other info.

Regards,
Uday

Comment 11 Rich Megginson 2015-04-07 14:22:19 UTC
(In reply to Udayendu Kar from comment #10)
> # rpm -qa | grep -i selinux
> rpm-plugin-selinux-4.12.0.1-5.fc21.x86_64
> selinux-policy-3.13.1-105.9.fc21.noarch
> libselinux-ruby-2.3-5.fc21.x86_64
> libselinux-python-2.3-5.fc21.x86_64
> selinux-policy-targeted-3.13.1-105.9.fc21.noarch
> libselinux-utils-2.3-5.fc21.x86_64
> libselinux-2.3-5.fc21.x86_64
> 
> 
> # cat /etc/redhat-release 
> Fedora release 21 (Twenty One)
> 
> # uname -a
> Linux localhost.localdomain 3.19.3-200.fc21.x86_64 #1 SMP Thu Mar 26
> 21:39:42 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
> 
> I also didnt get any clue in the audit logs and thats the reason opened this
> bz. This issue is 100% reproducible. So if you want you can try to reproduce
> it.
> 
> Let me know if you need any other info.
> 
> Regards,
> Uday

Yes, I can reproduce the problem.  However, my audit log looks completely different than yours.  I have a lot of httpd related AVCs in my audit log, you do not.  How do you explain this?

Comment 12 Udayendu Kar 2015-04-07 14:54:14 UTC
Strange !!

I have no clue. Which OS you have used ? I have used the F21 Server edition OS.

But as its an issue so may be based on your logs you can fix it.

Comment 13 Rich Megginson 2015-04-07 15:09:58 UTC
(In reply to Udayendu Kar from comment #12)
> Strange !!
> 
> I have no clue. Which OS you have used ? I have used the F21 Server edition
> OS.

I am using F21 Workstation (fedora-release-workstation-21-2).

> 
> But as its an issue so may be based on your logs you can fix it.

Ok, but that's no guarantee that it will fix your issue . . .

Comment 14 Lars Kellogg-Stedman 2015-04-07 15:17:18 UTC
I am confused, because in the original description you indicated that putting selinux in permissive mode worked around the problem...but your audit log does not show any selinux denials.

Are there any additional errors from httpd in either /var/log/httpd/error.log orin "journalctl -u httpd"?

Comment 15 Udayendu Kar 2015-04-08 13:24:28 UTC
Not sure Lars.

I will make a new setup as its 100% reproducible and then will let you know.

Comment 16 Udayendu Kar 2015-04-22 10:47:22 UTC
Hi,

Today I reproduced it on a F20 OS and able to reproduce it successfully with the same error message:

=== Error ====
ERROR : Error appeared during Puppet run: 192.168.122.60_keystone.pp
Error: /Stage[main]/Apache::Service/Service[httpd]: Failed to call refresh: Could not start Service[httpd]: Execution of '/sbin/service httpd start' returned 1: Redirecting to /bin/systemctl start  httpd.service
You will find full trace in log /var/tmp/packstack/20150422-060039-8tSJCV/manifests/192.168.122.60_keystone.pp.log
Please check log file /var/tmp/packstack/20150422-060039-8tSJCV/openstack-setup.log for more information
Additional information:
 * A new answerfile was created in: /root/packstack-answers-20150422-060040.txt
 * Time synchronization installation was skipped. Please note that unsynchronized time on server instances might be problem for some OpenStack components.
 * File /root/keystonerc_admin has been created on OpenStack client host 192.168.122.60. To use the command line tools you need to source the file.
 * To access the OpenStack Dashboard browse to http://192.168.122.60/dashboard .
Please, find your login credentials stored in the keystonerc_admin in your home directory.
 * To use Nagios, browse to http://192.168.122.60/nagios username: nagiosadmin, password: 7befd15cecd241e5
[root@localhost ~]# /bin/systemctl status  httpd.service
httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled)
   Active: failed (Result: exit-code) since Wed 2015-04-22 06:02:33 EDT; 38min ago
 Main PID: 4146 (code=exited, status=1/FAILURE)

Apr 22 06:02:33 localhost.localdomain httpd[4146]: (13)Permission denied: AH00072: make_sock: could not bind ...5000
Apr 22 06:02:33 localhost.localdomain httpd[4146]: (13)Permission denied: AH00072: make_sock: could not bind ...5000
Apr 22 06:02:33 localhost.localdomain httpd[4146]: no listening sockets available, shutting down
Apr 22 06:02:33 localhost.localdomain httpd[4146]: AH00015: Unable to open logs
Apr 22 06:02:33 localhost.localdomain systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE
Apr 22 06:02:33 localhost.localdomain systemd[1]: Failed to start The Apache HTTP Server.
Apr 22 06:02:33 localhost.localdomain systemd[1]: Unit httpd.service entered failed state.
Hint: Some lines were ellipsized, use -l to show in full.
======

I am also attaching the audit.log file for your refernce but again I am not able to see any avc related messages there. Not sure why but my auditd.service is running.

# systemctl status auditd.service 
auditd.service - Security Auditing Service
   Loaded: loaded (/usr/lib/systemd/system/auditd.service; enabled)
   Active: active (running) since Wed 2015-04-22 05:59:12 EDT; 46min ago
 Main PID: 515 (auditd)
   CGroup: /system.slice/auditd.service
           ├─515 /sbin/auditd -n
           ├─521 /sbin/audispd
           └─522 /usr/sbin/sedispatch

Apr 22 05:59:12 localhost.localdomain auditctl[516]: No rules
Apr 22 05:59:12 localhost.localdomain auditctl[516]: enabled 0
Apr 22 05:59:12 localhost.localdomain auditctl[516]: flag 1
Apr 22 05:59:12 localhost.localdomain auditctl[516]: pid 0
Apr 22 05:59:12 localhost.localdomain auditctl[516]: rate_limit 0
Apr 22 05:59:12 localhost.localdomain auditctl[516]: backlog_limit 320
Apr 22 05:59:12 localhost.localdomain auditctl[516]: lost 0
Apr 22 05:59:12 localhost.localdomain auditctl[516]: backlog 0
Apr 22 05:59:12 localhost.localdomain auditctl[516]: backlog_wait_time 60000
Apr 22 06:06:38 localhost.localdomain auditd[515]: Audit daemon rotating log files

Let me know if I am missing something here.

Comment 17 Udayendu Kar 2015-04-22 10:48:52 UTC
Created attachment 1017364 [details]
Latest audit logs on Fedora 20

Comment 18 Udayendu Kar 2015-04-22 10:50:06 UTC
Here are the httpd messages:

==--==
[Wed Apr 22 06:02:03.883828 2015] [core:notice] [pid 2879] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Wed Apr 22 06:02:03.904152 2015] [suexec:notice] [pid 2879] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Wed Apr 22 06:02:03.910123 2015] [auth_digest:notice] [pid 2879] AH01757: generating secret for digest authentication ...
[Wed Apr 22 06:02:03.913455 2015] [mpm_prefork:notice] [pid 2879] AH00163: Apache/2.4.10 (Fedora) configured -- resuming normal operations
[Wed Apr 22 06:02:03.913488 2015] [core:notice] [pid 2879] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Wed Apr 22 06:02:32.290316 2015] [mpm_prefork:notice] [pid 2879] AH00170: caught SIGWINCH, shutting down gracefully
==--==

Comment 19 Rich Megginson 2015-04-22 14:01:07 UTC
(In reply to Udayendu Kar from comment #17)
> Created attachment 1017364 [details]
> Latest audit logs on Fedora 20

Can you confirm that you can start httpd successfully if you disable selinux?

Comment 20 Udayendu Kar 2015-04-23 12:24:15 UTC
Hi Rich,

Yes, after setting the selinux into permissive mode I can start the httpd service properly:

[root@localhost ~]# systemctl status httpd.service
httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled)
   Active: failed (Result: exit-code) since Thu 2015-04-23 08:20:26 EDT; 1min 33s ago
  Process: 719 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=0/SUCCESS)
  Process: 588 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, status=1/FAILURE)
 Main PID: 588 (code=exited, status=1/FAILURE)
   CGroup: /system.slice/httpd.service

Apr 23 08:20:26 localhost.localdomain systemd[1]: Unit httpd.service entered failed state.
Apr 23 08:20:26 localhost.localdomain httpd[588]: (13)Permission denied: AH00072: make_sock: could not bind to address [::]:5000
Apr 23 08:20:26 localhost.localdomain httpd[588]: (13)Permission denied: AH00072: make_sock: could not bind to address 0.0.0.0:5000
Apr 23 08:20:26 localhost.localdomain httpd[588]: no listening sockets available, shutting down
Apr 23 08:20:26 localhost.localdomain httpd[588]: AH00015: Unable to open logs

[root@localhost ~]# systemctl start httpd.service
Job for httpd.service failed. See 'systemctl status httpd.service' and 'journalctl -xn' for details.

[root@localhost ~]# getenforce 
Enforcing

[root@localhost ~]# setenforce 0

[root@localhost ~]# getenforce 
Permissive

[root@localhost ~]# systemctl start httpd.service

[root@localhost ~]# systemctl status httpd.service
httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled)
   Active: active (running) since Thu 2015-04-23 08:22:20 EDT; 5s ago
  Process: 3021 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=0/SUCCESS)
 Main PID: 3050 (httpd)
   Status: "Processing requests..."
   CGroup: /system.slice/httpd.service
           ├─3050 /usr/sbin/httpd -DFOREGROUND
           ├─3059 /usr/sbin/httpd -DFOREGROUND
           ├─3060 /usr/sbin/httpd -DFOREGROUND
           ├─3061 /usr/sbin/httpd -DFOREGROUND
           ├─3062 /usr/sbin/httpd -DFOREGROUND
           ├─3063 /usr/sbin/httpd -DFOREGROUND
           ├─3064 /usr/sbin/httpd -DFOREGROUND
           ├─3065 /usr/sbin/httpd -DFOREGROUND
           ├─3066 /usr/sbin/httpd -DFOREGROUND
           ├─3067 /usr/sbin/httpd -DFOREGROUND
           └─3068 /usr/sbin/httpd -DFOREGROUND

Apr 23 08:22:20 localhost.localdomain systemd[1]: Started The Apache HTTP Server.
Apr 23 08:22:20 localhost.localdomain python[3052]: SELinux is preventing /usr/sbin/httpd from name_bind access on the tcp_socket .
                                                    
                                                    *****  Plugin catchall (100. confidence) suggests   **************************...
Hint: Some lines were ellipsized, use -l to show in full.


Let me know if you need any logs for further analysis.

Comment 21 Rich Megginson 2015-04-23 14:19:19 UTC
Ok.  I've contacted the selinux people.

Comment 22 Milos Malik ♈🏡🍅 2015-04-23 14:25:36 UTC
Please attach the SELinux denials:

# ausearch -m avc -m user_avc -m selinux_err -i -ts today

Comment 23 Udayendu Kar 2015-04-25 11:32:44 UTC
# ausearch -m avc -m user_avc -m selinux_err -i -ts today
----
type=PROCTITLE msg=audit(Saturday 25 April 2015 07:28:40.944:34) : proctitle=/usr/sbin/httpd -DFOREGROUND 
type=SYSCALL msg=audit(Saturday 25 April 2015 07:28:40.944:34) : arch=x86_64 syscall=bind success=no exit=-13(Permission denied) a0=0x6 a1=0x7f36d899b3a8 a2=0x1c a3=0x7ffc661f842c items=0 ppid=1 pid=590 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=httpd exe=/usr/sbin/httpd subj=system_u:system_r:httpd_t:s0 key=(null) 
type=AVC msg=audit(Saturday 25 April 2015 07:28:40.944:34) : avc:  denied  { name_bind } for  pid=590 comm=httpd src=5000 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:commplex_main_port_t:s0 tclass=tcp_socket permissive=0 
----
type=PROCTITLE msg=audit(Saturday 25 April 2015 07:28:40.945:35) : proctitle=/usr/sbin/httpd -DFOREGROUND 
type=SYSCALL msg=audit(Saturday 25 April 2015 07:28:40.945:35) : arch=x86_64 syscall=bind success=no exit=-13(Permission denied) a0=0x5 a1=0x7f36d899b2e8 a2=0x10 a3=0x7ffc661f842c items=0 ppid=1 pid=590 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=httpd exe=/usr/sbin/httpd subj=system_u:system_r:httpd_t:s0 key=(null) 
type=AVC msg=audit(Saturday 25 April 2015 07:28:40.945:35) : avc:  denied  { name_bind } for  pid=590 comm=httpd src=5000 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:commplex_main_port_t:s0 tclass=tcp_socket permissive=0 
----
type=USER_AVC msg=audit(Saturday 25 April 2015 07:28:57.969:70) : pid=610 uid=dbus auid=unset ses=unset subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc:  denied  { send_msg } for msgtype=method_return dest=:1.35 spid=609 tpid=2075 scontext=system_u:system_r:systemd_logind_t:s0 tcontext=system_u:system_r:cinder_volume_t:s0 tclass=dbus  exe=/usr/bin/dbus-daemon sauid=dbus hostname=? addr=? terminal=?' 
----
type=USER_AVC msg=audit(Saturday 25 April 2015 07:28:58.009:72) : pid=610 uid=dbus auid=unset ses=unset subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc:  denied  { send_msg } for msgtype=method_return dest=:1.36 spid=609 tpid=2077 scontext=system_u:system_r:systemd_logind_t:s0 tcontext=system_u:system_r:cinder_backup_t:s0 tclass=dbus  exe=/usr/bin/dbus-daemon sauid=dbus hostname=? addr=? terminal=?' 
----
type=USER_AVC msg=audit(Saturday 25 April 2015 07:29:23.191:95) : pid=610 uid=dbus auid=unset ses=unset subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc:  denied  { send_msg } for msgtype=method_return dest=:1.51 spid=609 tpid=2210 scontext=system_u:system_r:systemd_logind_t:s0 tcontext=system_u:system_r:cinder_volume_t:s0 tclass=dbus  exe=/usr/bin/dbus-daemon sauid=dbus hostname=? addr=? terminal=?' 
----
type=USER_AVC msg=audit(Saturday 25 April 2015 07:29:23.195:96) : pid=610 uid=dbus auid=unset ses=unset subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc:  denied  { send_msg } for msgtype=method_return dest=:1.52 spid=609 tpid=2209 scontext=system_u:system_r:systemd_logind_t:s0 tcontext=system_u:system_r:cinder_backup_t:s0 tclass=dbus  exe=/usr/bin/dbus-daemon sauid=dbus hostname=? addr=? terminal=?' 
----
type=USER_AVC msg=audit(Saturday 25 April 2015 07:29:48.316:110) : pid=610 uid=dbus auid=unset ses=unset subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc:  denied  { send_msg } for msgtype=method_return dest=:1.53 spid=609 tpid=2226 scontext=system_u:system_r:systemd_logind_t:s0 tcontext=system_u:system_r:cinder_volume_t:s0 tclass=dbus  exe=/usr/bin/dbus-daemon sauid=dbus hostname=? addr=? terminal=?' 
----
type=USER_AVC msg=audit(Saturday 25 April 2015 07:29:48.339:113) : pid=610 uid=dbus auid=unset ses=unset subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc:  denied  { send_msg } for msgtype=method_return dest=:1.54 spid=609 tpid=2229 scontext=system_u:system_r:systemd_logind_t:s0 tcontext=system_u:system_r:cinder_backup_t:s0 tclass=dbus  exe=/usr/bin/dbus-daemon sauid=dbus hostname=? addr=? termi

Comment 24 Milos Malik ♈🏡🍅 2015-04-27 11:00:26 UTC
The first 2 AVCs are fixed in selinux-policy-targeted-3.13.1-24.el7.noarch. The rest of USER_AVCs is nor allowed neither dontaudited:

# sesearch -s systemd_logind_t -t cinder_backup_t -c dbus -p send_msg -A -C

# sesearch -s systemd_logind_t -t cinder_volume_t -c dbus -p send_msg -A -C

#

Comment 25 Rich Megginson 2015-04-27 14:22:51 UTC
(In reply to Milos Malik from comment #24)
> The first 2 AVCs are fixed in selinux-policy-targeted-3.13.1-24.el7.noarch.

The bug reporter is using Fedora 21.  Is there a selinux-policy-targeted package for Fedora 21 that fixes this problem?

> The rest of USER_AVCs is nor allowed neither dontaudited:
> 
> # sesearch -s systemd_logind_t -t cinder_backup_t -c dbus -p send_msg -A -C
> 
> # sesearch -s systemd_logind_t -t cinder_volume_t -c dbus -p send_msg -A -C
> 
> #

Comment 26 Udayendu Kar 2015-04-27 14:57:24 UTC
Hi Milos,

There is no out put in the system for the above two search commands as mentioned in comments#24.

Also I would like to inform you that this issue is reproducible on both F20 and on F21. So selinux-policy-targeted package is require for both in F20 and F21.

Thanks,
Uday

Comment 28 Migi 2015-07-05 11:32:23 UTC
Description of problem:

Version-Release number of selected component (if applicable):
Fedora 21 / server / x86 64bit.
packstack Kilo 2015.1.dev1537.gba5183c
Apache/2.4.12 (Fedora)

How reproducible:

Every time running $packstack --allinone.
Trying to start httpd service manually fails as well.

Steps to Reproduce:
Trying to install openstack using $packstack --allinone on Fedora b21 x64.

1. Fedora was installed from dvd iso in vbox:
  Fedora-Server-DVD-x86_64-21.iso

2. NetworkManager was disabled and network configured using network service as in the instructions.

3. Followed all other steps from the https://www.rdoproject.org/Quickstart up to the point where command "$packstack --allinone" failed. 

Note that I was using root, so sudo is not required.

4. Trying to start httpd manually fails with informaiton that can not bind to address 0.0.0.0:80 no listening sockets available.

Checking what processes may occupy those ports does not give list of processes that  (lsof -i / netstat - plant).


Actual results:
Installation fails.


Expected results:
Installation passes as it's allinone setup.

Additional info:
It is not the issue with some process occupying the port. Changing the port to other number also gives the issue of 

Jul 05 12:34:02 localhost.localdomain httpd[19627]: (13)Permission denied: AH00072: make_sock: could not bind to address [::]:5030
Jul 05 12:34:02 localhost.localdomain httpd[19627]: (13)Permission denied: AH00072: make_sock: could not bind to address 0.0.0.0:5030

Because changing the port does not solve the issue (and no processes were actually using them) I vote there is problem with other audit preventing to open port by process. Quickly looking (I'm new to Fedora) at it reveals it may be a problem with selinux.

Following instructions from:

http://docs.fedoraproject.org/en-US/Fedora/13/html/Managing_Confined_Services/chap-Managing_Confined_Services-The_Apache_HTTP_Server.html

Installing semanage util and trying to add the port 5000 to the httpd process gives the following error:


I've managed to fix the issue by:
   $ yum -y install policycoreutils-python
   $ semanage port -a -t http_port_t -p tcp 5000
   ValueError: Port tcp/5000 already defined

   Listening port 5000:
   $ semanage port -l | grep 5000 |grep tcp
   complex_main_port_t   tcp   5000

From here it looks like http_port_t is not listed as port 5000

The work-around to start httpd I used was to:
   $ semanage port -a -t http_port_t -p tcp 5030
   > Change port to 5030 in both /etc/httpd/conf/ports.conf and /etc/httpd/conf.d/10-keystone_wsgi_main.conf
   
Now I was able to start httpd service.
   $ service httpd start

Comment 29 Migi 2015-07-05 11:41:00 UTC
I think the following are duplicates of the issue:

 #1174749
 #1209206

Comment 30 Lars Kellogg-Stedman 2015-08-04 13:36:02 UTC
I've confirmed this behavior on Fedora 21.  I still haven't achieved a functional install, but I needed *at least* the following:

    setsebool httpd_use_openstack true
    setsebool httpd_can_network_connect true
    setsebool httpd_can_network_connect_db true

I also need to add a policy exception for:

type=USER_AVC msg=audit(08/03/2015 23:37:25.674:49871) : pid=308 uid=dbus auid=unset ses=unset subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc:  denied  { send_msg } for msgtype=method_return dest=:1.3752 spid=307 tpid=18515 scontext=system_u:system_r:systemd_logind_t:s0 tcontext=system_u:system_r:cinder_volume_t:s0 tclass=dbus  exe=/usr/bin/dbus-daemon (deleted) sauid=dbus hostname=? addr=? terminal=?' 

Which translates into:

    module cinder_notify_systemd 1.0;

    require {
      type systemd_logind_t;
      type cinder_volume_t;
      class dbus send_msg;
    }

    #============= systemd_logind_t ==============
    allow systemd_logind_t cinder_volume_t:dbus send_msg;

Comment 32 Dustin Schoenbrun 2016-05-16 19:06:55 UTC
I know this is an old bug, but I was able to replicate this bug in CentOS 7.1 with the centos-release-openstack-mitaka packages and packstack. Setting SELinux to "permissive" allows the httpd service to start up properly.

Comment 33 Milos Malik ♈🏡🍅 2016-05-17 06:44:11 UTC
Does your scenario trigger any SELinux denials? Following command will help you to find out:

# ausearch -m avc,user_avc,selinux_err,user_selinux_err -i -ts today

If there are any SELinux denials, could you collect them and attach them here?

Comment 34 Chandan Kumar 2016-05-19 15:50:38 UTC
This bug is against a Version which has reached End of Life.
If it's still present in supported release (http://releases.openstack.org), please update Version and reopen.

Comment 35 godfrey_tan2001 2018-06-05 07:34:51 UTC
Hi,

I am getting these errors when installing Packstack --allinone.  The error relates to rabbit-mq server.  Here is part of the openstack install log file.

cd /usr/share/openstack-puppet/modules
tar --dereference -cpzf - aodh apache ceilometer certmonger cinder concat firewall glance gnocchi heat horizon inifile ironic keystone magnum manila memcached mysql neutron nova nssdb openstack openstacklib oslo ovn packstack panko rabbitmq redis remote rsync sahara ssh stdlib swift sysctl tempest trove vcsrepo vswitch xinetd | ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root@192.168.1.240 tar -C /var/tmp/packstack/659be2e3ef624c13a9ef023b95115ae8/modules -xpzf -
2018-06-05 15:21:33::ERROR::run_setup::1059::root:: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/packstack/installer/run_setup.py", line 1054, in main
    _main(options, confFile, logFile)
  File "/usr/lib/python2.7/site-packages/packstack/installer/run_setup.py", line 678, in _main
    runSequences()
  File "/usr/lib/python2.7/site-packages/packstack/installer/run_setup.py", line 645, in runSequences
    controller.runAllSequences()
  File "/usr/lib/python2.7/site-packages/packstack/installer/setup_controller.py", line 81, in runAllSequences
    sequence.run(config=self.CONF, messages=self.MESSAGES)
  File "/usr/lib/python2.7/site-packages/packstack/installer/core/sequences.py", line 109, in run
    step.run(config=config, messages=messages)
  File "/usr/lib/python2.7/site-packages/packstack/installer/core/sequences.py", line 50, in run
    self.function(config, messages)
  File "/usr/lib/python2.7/site-packages/packstack/plugins/puppet_950.py", line 214, in apply_puppet_manifest
    wait_for_puppet(currently_running, messages)
  File "/usr/lib/python2.7/site-packages/packstack/plugins/puppet_950.py", line 128, in wait_for_puppet
    validate_logfile(log)
  File "/usr/lib/python2.7/site-packages/packstack/modules/puppet.py", line 107, in validate_logfile
    raise PuppetError(message)
PuppetError: Error appeared during Puppet run: 192.168.1.240_controller.pp
Error: Systemd start for rabbitmq-server failed!
You will find full trace in log /var/tmp/packstack/20180605-151611-4E_RV9/manifests/192.168.1.240_controller.pp.log


How can I resolve this?

Thank you.


Note You need to log in before you can comment on or make changes to this bug.