Red Hat Bugzilla – Bug 1280083
rhel-osp-director: 8.0 - fail to deploy the undercloud due to puppet error "Could not start Service[ironic-inspector-dnsmasq]"
Last modified: 2016-04-07 17:11:19 EDT
rhel-osp-director: 8.0 - fail to deploy the undercloud due to puppet error "Could not start Service[ironic-inspector-dnsmasq]" Environment: instack-undercloud-2.1.3-1.el7ost.noarch Steps to reproduce: Attempt to deploy the undercloud. Result: Puppet error: Error: Could not start Service[ironic-inspector-dnsmasq]: Execution of '/bin/systemctl start openstack-ironic-inspector-dnsmasq' returned 1: Job for openstack-ironic-inspector-dnsmasq.service failed because the control process exited with error code. See "systemctl status openstack-ironic-inspector-dnsmasq.service" and "journalctl -xe" for details. Wrapped exception: Execution of '/bin/systemctl start openstack-ironic-inspector-dnsmasq' returned 1: Job for openstack-ironic-inspector-dnsmasq.service failed because the control process exited with error code. See "systemctl status openstack-ironic-inspector-dnsmasq.service" and "journalctl -xe" for details. Error: /Stage[main]/Ironic::Inspector/Service[ironic-inspector-dnsmasq]/ensure: change from stopped to running failed: Could not start Service[ironic-inspector-dnsmasq]: Execution of '/bin/systemctl start openstack-ironic-inspector-dnsmasq' returned 1: Job for openstack-ironic-inspector-dnsmasq.service failed because the control process exited with error code. See "systemctl status openstack-ironic-inspector-dnsmasq.service" and "journalctl -xe" for details. Then the deployment fails with: [2015-11-10 17:11:38,536] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 6] [2015-11-10 17:11:38,537] (os-refresh-config) [ERROR] Aborting... Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 562, in install _run_orc(instack_env) File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 494, in _run_orc _run_live_command(args, instack_env, 'os-refresh-config') File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 325, in _run_live_command raise RuntimeError('%s failed. See log for details.' % name) RuntimeError: os-refresh-config failed. See log for details. Command 'instack-install-undercloud' returned non-zero exit status 1 Expected result: The undercloud deployment should complete with no errors.
Created attachment 1092480 [details] /var/log/messages
selinux issue, was abe to start openstack-ironic-inspector-dnsmasq only after running "setenforce 0"
Created attachment 1092501 [details] /var/log/audit/audit.log
type=AVC msg=audit(1447193372.173:336): avc: denied { read } for pid=25420 comm="dnsmasq" name="tftpboot" dev="sda1" ino=1572866 scontext=system_u:system_r:dnsmasq_t:s0 tcontext=unconfined_u:object_r:default_t:s0 tclass=dir The above is caused because a dir is being created in the '/' directory that does not have the correct label. 'default_t' is assigned to a directory in '/' that hasn't had it's label restored or assigned. The fix for this is after you create '/<my_dir' run $ restorecon -Rv /<my_dir> . If that doesn't work we can assign a context to it. Adding this rule: allow neutron_t httpd_config_t:dir search;
Are these logs from permissive mode? Permissive will capture all the AVC instead of the first failure.
The logs include the original state (enforced) and permissive afterwards.
On the system with issues ran: "ls -Z /|grep default" Result: drwxr-xr-x. ironic ironic unconfined_u:object_r:default_t:s0 tftpboot Then ran: "sudo restorecon -Rv /tftpboot/" ls -Z / |grep tftpboot drwxr-xr-x. ironic ironic unconfined_u:object_r:tftpdir_t:s0 tftpboot
openstack-selinux-0.6.43-1.el7ost and the above restorecon will do the trick.
I am pretty sure this puppet-ironic patch will resolve the type context on the tftpboot dir without a need to run restorecon: https://review.openstack.org/244675 If not, the fix would be in the instack-undercloud puppet-stack-config element.
Deploying undercloud still fails with the same error. Version: openstack-selinux-0.6.45-1.el7ost.noarch found 1 alerts in /var/log/audit/audit.log -------------------------------------------------------------------------------- SELinux is preventing /usr/sbin/dnsmasq from read access on the directory tftpboot. ***** Plugin catchall_labels (83.8 confidence) suggests ******************* If you want to allow dnsmasq to have read access on the tftpboot directory Then you need to change the label on tftpboot Do # semanage fcontext -a -t FILE_TYPE 'tftpboot' where FILE_TYPE is one of the following: NetworkManager_etc_t, NetworkManager_var_run_t, bin_t, boot_t, cert_t, cluster_conf_t, cluster_var_lib_t, cluster_var_run_t, crond_var_run_t, dbusd_etc_t, device_t, devpts_t, dnsmasq_etc_t, dnsmasq_lease_t, dnsmasq_var_run_t, etc_runtime_t, etc_t, file_context_t, fonts_cache_t, fonts_t, lib_t, locale_t, man_cache_t, man_t, net_conf_t, neutron_var_lib_t, nscd_var_run_t, proc_t, root_t, samba_etc_t, shell_exec_t, src_t, sssd_public_t, sysfs_t, system_conf_t, system_db_t, textrel_shlib_t, tftpdir_rw_t, tftpdir_t, udev_var_run_t, usr_t, var_lib_t, var_log_t, var_run_t, var_yp_t, virt_var_lib_t, virt_var_run_t. Then execute: restorecon -v 'tftpboot' ***** Plugin catchall (17.1 confidence) suggests ************************** If you believe that dnsmasq should be allowed read access on the tftpboot directory by default. Then you should report this as a bug. You can generate a local policy module to allow this access. Do allow this access for now by executing: # grep dnsmasq /var/log/audit/audit.log | audit2allow -M mypol # semodule -i mypol.pp Additional Information: Source Context system_u:system_r:dnsmasq_t:s0 Target Context unconfined_u:object_r:default_t:s0 Target Objects tftpboot [ dir ] Source dnsmasq Source Path /usr/sbin/dnsmasq Port <Unknown> Host <Unknown> Source RPM Packages dnsmasq-2.66-14.el7_1.x86_64 Target RPM Packages Policy RPM selinux-policy-3.13.1-60.el7.noarch Selinux Enabled True Policy Type targeted Enforcing Mode Enforcing Host Name instack.localdomain Platform Linux instack.localdomain 3.10.0-327.el7.x86_64 #1 SMP Thu Oct 29 17:29:29 EDT 2015 x86_64 x86_64 Alert Count 1 First Seen 2015-11-21 21:05:07 EST Last Seen 2015-11-21 21:05:07 EST Local ID b6b3f001-fa58-4835-98b2-98fd4848e37c Raw Audit Messages type=AVC msg=audit(1448157907.536:341): avc: denied { read } for pid=25534 comm="dnsmasq" name="tftpboot" dev="sda1" ino=524290 scontext=system_u:system_r:dnsmasq_t:s0 tcontext=unconfined_u:object_r:default_t:s0 tclass=dir type=SYSCALL msg=audit(1448157907.536:341): arch=x86_64 syscall=openat success=no exit=EACCES a0=ffffffffffffff9c a1=7f3afc17ef50 a2=90800 a3=0 items=0 ppid=1 pid=25534 auid=4294967295 uid=99 gid=40 euid=99 suid=99 fsuid=99 egid=40 sgid=40 fsgid=40 tty=(none) ses=4294967295 comm=dnsmasq exe=/usr/sbin/dnsmasq subj=system_u:system_r:dnsmasq_t:s0 key=(null) Hash: dnsmasq,dnsmasq_t,default_t,dir,read
This is the same error as before with 'default_t'. This needs to be run after tftpboot is created or right before `dnsmasq` is run. `sudo restorecon -Rv /tftpboot/`
I am also hitting this issue, was able to have a clean install after running following command: `sudo restorecon -Rv /tftpboot/`
While restorecon is one method to solve this, I think it is even better to just set the type context to tftpdir_t when we create the directory. That is the approach I took in https://review.openstack.org/244675 This approach has been verified to resolve the issue.
*** Bug 1283414 has been marked as a duplicate of this bug. ***
Upstream liberty backport: https://review.openstack.org/#/c/264180/
*** Bug 1296562 has been marked as a duplicate of this bug. ***
This will go into the next opm build, which should be 7.0.5
Hi, I can confirm that the patch mentioned in #16 works but unfortunately it stills not include in the last OSP-d 8 puddle. Thanks anyway! Regards,
(In reply to Alexandre Maumené from comment #22) > Hi, > > I can confirm that the patch mentioned in #16 works but unfortunately it > stills not include in the last OSP-d 8 puddle. > > Thanks anyway! > > Regards, This change is in the core puddle being generated today.l
*** Bug 1306859 has been marked as a duplicate of this bug. ***
Verified: Environment: openstack-puppet-modules-7.0.6-2.el7ost.noarch Successfully deployed the undercloud without disabling the selinux.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHEA-2016-0603.html