Bug 1280083 - rhel-osp-director: 8.0 - fail to deploy the undercloud due to puppet error "Could not start Service[ironic-inspector-dnsmasq]"
rhel-osp-director: 8.0 - fail to deploy the undercloud due to puppet error "C...
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-puppet-modules (Show other bugs)
8.0 (Liberty)
Unspecified Unspecified
urgent Severity high
: ga
: 8.0 (Liberty)
Assigned To: Emilien Macchi
Alexander Chuzhoy
: Automation, Triaged
: 1296562 1306859 (view as bug list)
Depends On:
Blocks: 1295992
  Show dependency treegraph
 
Reported: 2015-11-10 17:14 EST by Alexander Chuzhoy
Modified: 2016-04-07 17:11 EDT (History)
18 users (show)

See Also:
Fixed In Version: openstack-puppet-modules-7.0.6-1.el7ost
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-04-07 17:11:19 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
/var/log/messages (15.86 KB, application/x-gzip)
2015-11-10 17:16 EST, Alexander Chuzhoy
no flags Details
/var/log/audit/audit.log (86.50 KB, application/x-gzip)
2015-11-10 18:04 EST, Alexander Chuzhoy
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Gerrithub.io 259961 None None None 2016-01-22 11:17 EST
Launchpad 1515630 None None None Never
OpenStack gerrit 244675 None None None Never
OpenStack gerrit 264180 None None None 2016-01-06 08:51 EST

  None (edit)
Description Alexander Chuzhoy 2015-11-10 17:14:18 EST
rhel-osp-director: 8.0 - fail to deploy the undercloud due to puppet error "Could not start Service[ironic-inspector-dnsmasq]"


Environment:
instack-undercloud-2.1.3-1.el7ost.noarch

Steps to reproduce:
Attempt to deploy the undercloud.


Result:
Puppet error:

Error: Could not start Service[ironic-inspector-dnsmasq]: Execution of '/bin/systemctl start openstack-ironic-inspector-dnsmasq' returned 1: Job for openstack-ironic-inspector-dnsmasq.service failed because the control process exited with error code. See "systemctl status openstack-ironic-inspector-dnsmasq.service" and "journalctl -xe" for details.                                                                            
Wrapped exception:                                                                                                                                                                                                   
Execution of '/bin/systemctl start openstack-ironic-inspector-dnsmasq' returned 1: Job for openstack-ironic-inspector-dnsmasq.service failed because the control process exited with error code. See "systemctl status openstack-ironic-inspector-dnsmasq.service" and "journalctl -xe" for details.                                                                                                                                      
Error: /Stage[main]/Ironic::Inspector/Service[ironic-inspector-dnsmasq]/ensure: change from stopped to running failed: Could not start Service[ironic-inspector-dnsmasq]: Execution of '/bin/systemctl start openstack-ironic-inspector-dnsmasq' returned 1: Job for openstack-ironic-inspector-dnsmasq.service failed because the control process exited with error code. See "systemctl status openstack-ironic-inspector-dnsmasq.service" and "journalctl -xe" for details.


Then the deployment fails with:
[2015-11-10 17:11:38,536] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 6]

[2015-11-10 17:11:38,537] (os-refresh-config) [ERROR] Aborting...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 562, in install
    _run_orc(instack_env)
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 494, in _run_orc
    _run_live_command(args, instack_env, 'os-refresh-config')
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 325, in _run_live_command
    raise RuntimeError('%s failed. See log for details.' % name)
RuntimeError: os-refresh-config failed. See log for details.
Command 'instack-install-undercloud' returned non-zero exit status 1


Expected result:

The undercloud deployment should complete with no errors.
Comment 2 Alexander Chuzhoy 2015-11-10 17:16 EST
Created attachment 1092480 [details]
/var/log/messages
Comment 3 Alexander Chuzhoy 2015-11-10 17:29:56 EST
selinux issue,
was abe to start openstack-ironic-inspector-dnsmasq only after running "setenforce 0"
Comment 4 Alexander Chuzhoy 2015-11-10 18:04 EST
Created attachment 1092501 [details]
/var/log/audit/audit.log
Comment 6 Ryan Hallisey 2015-11-11 12:48:04 EST
type=AVC msg=audit(1447193372.173:336): avc: denied { read } for pid=25420 comm="dnsmasq" name="tftpboot" dev="sda1" ino=1572866 scontext=system_u:system_r:dnsmasq_t:s0 tcontext=unconfined_u:object_r:default_t:s0 tclass=dir

The above is caused because a dir is being created in the '/' directory that does not have the correct label.  'default_t' is assigned to a directory in '/' that hasn't had it's label restored or assigned.  The fix for this is after you create '/<my_dir' run $ restorecon -Rv /<my_dir> .  If that doesn't work we can assign a context to it.

Adding this rule:
allow neutron_t httpd_config_t:dir search;
Comment 7 Ryan Hallisey 2015-11-11 12:49:05 EST
Are these logs from permissive mode?  Permissive will capture all the AVC instead of the first failure.
Comment 8 Alexander Chuzhoy 2015-11-11 12:54:51 EST
The logs include the original state (enforced) and permissive afterwards.
Comment 9 Alexander Chuzhoy 2015-11-11 13:11:04 EST
On the system with issues ran: "ls -Z /|grep default"
Result:
drwxr-xr-x. ironic ironic unconfined_u:object_r:default_t:s0 tftpboot


Then ran:
"sudo restorecon -Rv /tftpboot/"

ls -Z / |grep tftpboot
drwxr-xr-x. ironic ironic unconfined_u:object_r:tftpdir_t:s0 tftpboot
Comment 10 Ryan Hallisey 2015-11-11 14:11:44 EST
openstack-selinux-0.6.43-1.el7ost and the above restorecon will do the trick.
Comment 11 John Trowbridge 2015-11-12 09:14:45 EST
I am pretty sure this puppet-ironic patch will resolve the type context on the tftpboot dir without a need to run restorecon:

https://review.openstack.org/244675

If not, the fix would be in the instack-undercloud puppet-stack-config element.
Comment 13 Amit Ugol 2015-11-21 21:29:47 EST
Deploying undercloud still fails with the same error.
Version: openstack-selinux-0.6.45-1.el7ost.noarch


found 1 alerts in /var/log/audit/audit.log
--------------------------------------------------------------------------------

SELinux is preventing /usr/sbin/dnsmasq from read access on the directory tftpboot.

*****  Plugin catchall_labels (83.8 confidence) suggests   *******************

If you want to allow dnsmasq to have read access on the tftpboot directory
Then you need to change the label on tftpboot
Do
# semanage fcontext -a -t FILE_TYPE 'tftpboot'
where FILE_TYPE is one of the following: NetworkManager_etc_t, NetworkManager_var_run_t, bin_t, boot_t, cert_t, cluster_conf_t, cluster_var_lib_t, cluster_var_run_t, crond_var_run_t, dbusd_etc_t, device_t, devpts_t, dnsmasq_etc_t, dnsmasq_lease_t, dnsmasq_var_run_t, etc_runtime_t, etc_t, file_context_t, fonts_cache_t, fonts_t, lib_t, locale_t, man_cache_t, man_t, net_conf_t, neutron_var_lib_t, nscd_var_run_t, proc_t, root_t, samba_etc_t, shell_exec_t, src_t, sssd_public_t, sysfs_t, system_conf_t, system_db_t, textrel_shlib_t, tftpdir_rw_t, tftpdir_t, udev_var_run_t, usr_t, var_lib_t, var_log_t, var_run_t, var_yp_t, virt_var_lib_t, virt_var_run_t. 
Then execute: 
restorecon -v 'tftpboot'


*****  Plugin catchall (17.1 confidence) suggests   **************************

If you believe that dnsmasq should be allowed read access on the tftpboot directory by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# grep dnsmasq /var/log/audit/audit.log | audit2allow -M mypol
# semodule -i mypol.pp


Additional Information:
Source Context                system_u:system_r:dnsmasq_t:s0
Target Context                unconfined_u:object_r:default_t:s0
Target Objects                tftpboot [ dir ]
Source                        dnsmasq
Source Path                   /usr/sbin/dnsmasq
Port                          <Unknown>
Host                          <Unknown>
Source RPM Packages           dnsmasq-2.66-14.el7_1.x86_64
Target RPM Packages           
Policy RPM                    selinux-policy-3.13.1-60.el7.noarch
Selinux Enabled               True
Policy Type                   targeted
Enforcing Mode                Enforcing
Host Name                     instack.localdomain
Platform                      Linux instack.localdomain 3.10.0-327.el7.x86_64 #1
                              SMP Thu Oct 29 17:29:29 EDT 2015 x86_64 x86_64
Alert Count                   1
First Seen                    2015-11-21 21:05:07 EST
Last Seen                     2015-11-21 21:05:07 EST
Local ID                      b6b3f001-fa58-4835-98b2-98fd4848e37c

Raw Audit Messages
type=AVC msg=audit(1448157907.536:341): avc:  denied  { read } for  pid=25534 comm="dnsmasq" name="tftpboot" dev="sda1" ino=524290 scontext=system_u:system_r:dnsmasq_t:s0 tcontext=unconfined_u:object_r:default_t:s0 tclass=dir


type=SYSCALL msg=audit(1448157907.536:341): arch=x86_64 syscall=openat success=no exit=EACCES a0=ffffffffffffff9c a1=7f3afc17ef50 a2=90800 a3=0 items=0 ppid=1 pid=25534 auid=4294967295 uid=99 gid=40 euid=99 suid=99 fsuid=99 egid=40 sgid=40 fsgid=40 tty=(none) ses=4294967295 comm=dnsmasq exe=/usr/sbin/dnsmasq subj=system_u:system_r:dnsmasq_t:s0 key=(null)

Hash: dnsmasq,dnsmasq_t,default_t,dir,read
Comment 14 Ryan Hallisey 2015-11-23 08:36:25 EST
This is the same error as before with 'default_t'. This needs to be run after tftpboot is created or right before `dnsmasq` is run.
`sudo restorecon -Rv /tftpboot/`
Comment 15 Qasim Sarfraz 2015-12-02 10:17:49 EST
I am also hitting this issue, was able to have a clean install after running following command:
`sudo restorecon -Rv /tftpboot/`
Comment 16 John Trowbridge 2015-12-02 14:33:42 EST
While restorecon is one method to solve this, I think it is even better to just set the type context to tftpdir_t when we create the directory.

That is the approach I took in https://review.openstack.org/244675
This approach has been verified to resolve the issue.
Comment 18 Ryan Hallisey 2016-01-04 12:16:11 EST
*** Bug 1283414 has been marked as a duplicate of this bug. ***
Comment 19 John Trowbridge 2016-01-06 08:52:03 EST
Upstream liberty backport: https://review.openstack.org/#/c/264180/
Comment 20 Mike Burns 2016-01-07 10:35:43 EST
*** Bug 1296562 has been marked as a duplicate of this bug. ***
Comment 21 Jason Guiditta 2016-01-22 11:17:16 EST
This will go into the next opm build, which should be 7.0.5
Comment 22 Alexandre Maumené 2016-02-04 11:08:21 EST
Hi,

I can confirm that the patch mentioned in #16 works but unfortunately it stills not include in the last OSP-d 8 puddle.

Thanks anyway!

Regards,
Comment 24 Jon Schlueter 2016-02-04 13:41:43 EST
(In reply to Alexandre Maumené from comment #22)
> Hi,
> 
> I can confirm that the patch mentioned in #16 works but unfortunately it
> stills not include in the last OSP-d 8 puddle.
> 
> Thanks anyway!
> 
> Regards,

This change is in the core puddle being generated today.l
Comment 26 Mike Burns 2016-02-11 18:42:03 EST
*** Bug 1306859 has been marked as a duplicate of this bug. ***
Comment 27 Alexander Chuzhoy 2016-02-23 15:04:03 EST
Verified:

Environment:
openstack-puppet-modules-7.0.6-2.el7ost.noarch

Successfully deployed the undercloud without disabling the selinux.
Comment 29 errata-xmlrpc 2016-04-07 17:11:19 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-0603.html

Note You need to log in before you can comment on or make changes to this bug.