Bug 1280083 - rhel-osp-director: 8.0 - fail to deploy the undercloud due to puppet error "Could not start Service[ironic-inspector-dnsmasq]"
Summary: rhel-osp-director: 8.0 - fail to deploy the undercloud due to puppet error "C...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-puppet-modules
Version: 8.0 (Liberty)
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: ga
: 8.0 (Liberty)
Assignee: Emilien Macchi
QA Contact: Alexander Chuzhoy
URL:
Whiteboard:
: 1296562 1306859 (view as bug list)
Depends On:
Blocks: 1295992
TreeView+ depends on / blocked
 
Reported: 2015-11-10 22:14 UTC by Alexander Chuzhoy
Modified: 2016-04-07 21:11 UTC (History)
18 users (show)

Fixed In Version: openstack-puppet-modules-7.0.6-1.el7ost
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-04-07 21:11:19 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
/var/log/messages (15.86 KB, application/x-gzip)
2015-11-10 22:16 UTC, Alexander Chuzhoy
no flags Details
/var/log/audit/audit.log (86.50 KB, application/x-gzip)
2015-11-10 23:04 UTC, Alexander Chuzhoy
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Gerrithub.io 259961 0 None None None 2016-01-22 16:17:16 UTC
Launchpad 1515630 0 None None None Never
OpenStack gerrit 244675 0 None MERGED [inspector] Specify SELinux type context for tftp dir 2020-09-30 20:05:42 UTC
OpenStack gerrit 264180 0 None MERGED [inspector] Specify SELinux type context for tftp dir 2020-09-30 20:05:36 UTC
Red Hat Product Errata RHEA-2016:0603 0 normal SHIPPED_LIVE Red Hat OpenStack Platform 8 Enhancement Advisory 2016-04-08 00:53:53 UTC

Description Alexander Chuzhoy 2015-11-10 22:14:18 UTC
rhel-osp-director: 8.0 - fail to deploy the undercloud due to puppet error "Could not start Service[ironic-inspector-dnsmasq]"


Environment:
instack-undercloud-2.1.3-1.el7ost.noarch

Steps to reproduce:
Attempt to deploy the undercloud.


Result:
Puppet error:

Error: Could not start Service[ironic-inspector-dnsmasq]: Execution of '/bin/systemctl start openstack-ironic-inspector-dnsmasq' returned 1: Job for openstack-ironic-inspector-dnsmasq.service failed because the control process exited with error code. See "systemctl status openstack-ironic-inspector-dnsmasq.service" and "journalctl -xe" for details.                                                                            
Wrapped exception:                                                                                                                                                                                                   
Execution of '/bin/systemctl start openstack-ironic-inspector-dnsmasq' returned 1: Job for openstack-ironic-inspector-dnsmasq.service failed because the control process exited with error code. See "systemctl status openstack-ironic-inspector-dnsmasq.service" and "journalctl -xe" for details.                                                                                                                                      
Error: /Stage[main]/Ironic::Inspector/Service[ironic-inspector-dnsmasq]/ensure: change from stopped to running failed: Could not start Service[ironic-inspector-dnsmasq]: Execution of '/bin/systemctl start openstack-ironic-inspector-dnsmasq' returned 1: Job for openstack-ironic-inspector-dnsmasq.service failed because the control process exited with error code. See "systemctl status openstack-ironic-inspector-dnsmasq.service" and "journalctl -xe" for details.


Then the deployment fails with:
[2015-11-10 17:11:38,536] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 6]

[2015-11-10 17:11:38,537] (os-refresh-config) [ERROR] Aborting...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 562, in install
    _run_orc(instack_env)
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 494, in _run_orc
    _run_live_command(args, instack_env, 'os-refresh-config')
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 325, in _run_live_command
    raise RuntimeError('%s failed. See log for details.' % name)
RuntimeError: os-refresh-config failed. See log for details.
Command 'instack-install-undercloud' returned non-zero exit status 1


Expected result:

The undercloud deployment should complete with no errors.

Comment 2 Alexander Chuzhoy 2015-11-10 22:16:50 UTC
Created attachment 1092480 [details]
/var/log/messages

Comment 3 Alexander Chuzhoy 2015-11-10 22:29:56 UTC
selinux issue,
was abe to start openstack-ironic-inspector-dnsmasq only after running "setenforce 0"

Comment 4 Alexander Chuzhoy 2015-11-10 23:04:47 UTC
Created attachment 1092501 [details]
/var/log/audit/audit.log

Comment 6 Ryan Hallisey 2015-11-11 17:48:04 UTC
type=AVC msg=audit(1447193372.173:336): avc: denied { read } for pid=25420 comm="dnsmasq" name="tftpboot" dev="sda1" ino=1572866 scontext=system_u:system_r:dnsmasq_t:s0 tcontext=unconfined_u:object_r:default_t:s0 tclass=dir

The above is caused because a dir is being created in the '/' directory that does not have the correct label.  'default_t' is assigned to a directory in '/' that hasn't had it's label restored or assigned.  The fix for this is after you create '/<my_dir' run $ restorecon -Rv /<my_dir> .  If that doesn't work we can assign a context to it.

Adding this rule:
allow neutron_t httpd_config_t:dir search;

Comment 7 Ryan Hallisey 2015-11-11 17:49:05 UTC
Are these logs from permissive mode?  Permissive will capture all the AVC instead of the first failure.

Comment 8 Alexander Chuzhoy 2015-11-11 17:54:51 UTC
The logs include the original state (enforced) and permissive afterwards.

Comment 9 Alexander Chuzhoy 2015-11-11 18:11:04 UTC
On the system with issues ran: "ls -Z /|grep default"
Result:
drwxr-xr-x. ironic ironic unconfined_u:object_r:default_t:s0 tftpboot


Then ran:
"sudo restorecon -Rv /tftpboot/"

ls -Z / |grep tftpboot
drwxr-xr-x. ironic ironic unconfined_u:object_r:tftpdir_t:s0 tftpboot

Comment 10 Ryan Hallisey 2015-11-11 19:11:44 UTC
openstack-selinux-0.6.43-1.el7ost and the above restorecon will do the trick.

Comment 11 John Trowbridge 2015-11-12 14:14:45 UTC
I am pretty sure this puppet-ironic patch will resolve the type context on the tftpboot dir without a need to run restorecon:

https://review.openstack.org/244675

If not, the fix would be in the instack-undercloud puppet-stack-config element.

Comment 13 Amit Ugol 2015-11-22 02:29:47 UTC
Deploying undercloud still fails with the same error.
Version: openstack-selinux-0.6.45-1.el7ost.noarch


found 1 alerts in /var/log/audit/audit.log
--------------------------------------------------------------------------------

SELinux is preventing /usr/sbin/dnsmasq from read access on the directory tftpboot.

*****  Plugin catchall_labels (83.8 confidence) suggests   *******************

If you want to allow dnsmasq to have read access on the tftpboot directory
Then you need to change the label on tftpboot
Do
# semanage fcontext -a -t FILE_TYPE 'tftpboot'
where FILE_TYPE is one of the following: NetworkManager_etc_t, NetworkManager_var_run_t, bin_t, boot_t, cert_t, cluster_conf_t, cluster_var_lib_t, cluster_var_run_t, crond_var_run_t, dbusd_etc_t, device_t, devpts_t, dnsmasq_etc_t, dnsmasq_lease_t, dnsmasq_var_run_t, etc_runtime_t, etc_t, file_context_t, fonts_cache_t, fonts_t, lib_t, locale_t, man_cache_t, man_t, net_conf_t, neutron_var_lib_t, nscd_var_run_t, proc_t, root_t, samba_etc_t, shell_exec_t, src_t, sssd_public_t, sysfs_t, system_conf_t, system_db_t, textrel_shlib_t, tftpdir_rw_t, tftpdir_t, udev_var_run_t, usr_t, var_lib_t, var_log_t, var_run_t, var_yp_t, virt_var_lib_t, virt_var_run_t. 
Then execute: 
restorecon -v 'tftpboot'


*****  Plugin catchall (17.1 confidence) suggests   **************************

If you believe that dnsmasq should be allowed read access on the tftpboot directory by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# grep dnsmasq /var/log/audit/audit.log | audit2allow -M mypol
# semodule -i mypol.pp


Additional Information:
Source Context                system_u:system_r:dnsmasq_t:s0
Target Context                unconfined_u:object_r:default_t:s0
Target Objects                tftpboot [ dir ]
Source                        dnsmasq
Source Path                   /usr/sbin/dnsmasq
Port                          <Unknown>
Host                          <Unknown>
Source RPM Packages           dnsmasq-2.66-14.el7_1.x86_64
Target RPM Packages           
Policy RPM                    selinux-policy-3.13.1-60.el7.noarch
Selinux Enabled               True
Policy Type                   targeted
Enforcing Mode                Enforcing
Host Name                     instack.localdomain
Platform                      Linux instack.localdomain 3.10.0-327.el7.x86_64 #1
                              SMP Thu Oct 29 17:29:29 EDT 2015 x86_64 x86_64
Alert Count                   1
First Seen                    2015-11-21 21:05:07 EST
Last Seen                     2015-11-21 21:05:07 EST
Local ID                      b6b3f001-fa58-4835-98b2-98fd4848e37c

Raw Audit Messages
type=AVC msg=audit(1448157907.536:341): avc:  denied  { read } for  pid=25534 comm="dnsmasq" name="tftpboot" dev="sda1" ino=524290 scontext=system_u:system_r:dnsmasq_t:s0 tcontext=unconfined_u:object_r:default_t:s0 tclass=dir


type=SYSCALL msg=audit(1448157907.536:341): arch=x86_64 syscall=openat success=no exit=EACCES a0=ffffffffffffff9c a1=7f3afc17ef50 a2=90800 a3=0 items=0 ppid=1 pid=25534 auid=4294967295 uid=99 gid=40 euid=99 suid=99 fsuid=99 egid=40 sgid=40 fsgid=40 tty=(none) ses=4294967295 comm=dnsmasq exe=/usr/sbin/dnsmasq subj=system_u:system_r:dnsmasq_t:s0 key=(null)

Hash: dnsmasq,dnsmasq_t,default_t,dir,read

Comment 14 Ryan Hallisey 2015-11-23 13:36:25 UTC
This is the same error as before with 'default_t'. This needs to be run after tftpboot is created or right before `dnsmasq` is run.
`sudo restorecon -Rv /tftpboot/`

Comment 15 Qasim Sarfraz 2015-12-02 15:17:49 UTC
I am also hitting this issue, was able to have a clean install after running following command:
`sudo restorecon -Rv /tftpboot/`

Comment 16 John Trowbridge 2015-12-02 19:33:42 UTC
While restorecon is one method to solve this, I think it is even better to just set the type context to tftpdir_t when we create the directory.

That is the approach I took in https://review.openstack.org/244675
This approach has been verified to resolve the issue.

Comment 18 Ryan Hallisey 2016-01-04 17:16:11 UTC
*** Bug 1283414 has been marked as a duplicate of this bug. ***

Comment 19 John Trowbridge 2016-01-06 13:52:03 UTC
Upstream liberty backport: https://review.openstack.org/#/c/264180/

Comment 20 Mike Burns 2016-01-07 15:35:43 UTC
*** Bug 1296562 has been marked as a duplicate of this bug. ***

Comment 21 Jason Guiditta 2016-01-22 16:17:16 UTC
This will go into the next opm build, which should be 7.0.5

Comment 22 Alexandre Maumené 2016-02-04 16:08:21 UTC
Hi,

I can confirm that the patch mentioned in #16 works but unfortunately it stills not include in the last OSP-d 8 puddle.

Thanks anyway!

Regards,

Comment 24 Jon Schlueter 2016-02-04 18:41:43 UTC
(In reply to Alexandre Maumené from comment #22)
> Hi,
> 
> I can confirm that the patch mentioned in #16 works but unfortunately it
> stills not include in the last OSP-d 8 puddle.
> 
> Thanks anyway!
> 
> Regards,

This change is in the core puddle being generated today.l

Comment 26 Mike Burns 2016-02-11 23:42:03 UTC
*** Bug 1306859 has been marked as a duplicate of this bug. ***

Comment 27 Alexander Chuzhoy 2016-02-23 20:04:03 UTC
Verified:

Environment:
openstack-puppet-modules-7.0.6-2.el7ost.noarch

Successfully deployed the undercloud without disabling the selinux.

Comment 29 errata-xmlrpc 2016-04-07 21:11:19 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-0603.html


Note You need to log in before you can comment on or make changes to this bug.