Bug 1378304
| Summary: | Failed to activate service locale and machine | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Douglas Schilling Landgraf <dougsland> |
| Component: | systemd | Assignee: | systemd-maint |
| Status: | CLOSED DUPLICATE | QA Contact: | qe-baseos-daemons |
| Severity: | urgent | Docs Contact: | |
| Priority: | high | ||
| Version: | 7.3 | CC: | bmcclain, danken, dougsland, fdeutsch, systemd-maint-list, ycui |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-10-04 16:45:32 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1369400 | ||
This looks like a some problem with the mount namespace. Could you try to remove the Private* and Protect* stanzas from unit-files in those services? And if that helps could you try to find out which of those is causing the issue. Also do you know what was the last version where this was working? We did not change anything in that area recently. Hi Lukáš, (In reply to Lukáš Nykrýn from comment #1) > This looks like a some problem with the mount namespace. Could you try to > remove the Private* and Protect* stanzas from unit-files in those services? > And if that helps could you try to find out which of those is causing the > issue. If I comment PrivateDevices=yes in the services: systemd-localed and systemd-machined everything work again (TUI and start virtual machines). I have tried several combinations and only PrivateDevices affect us. One thing that called my attention, from: https://www.freedesktop.org/software/systemd/man/systemd.exec.html <snip> PrivateDevices= Takes a boolean argument. If true, sets up a new /dev namespace for the executed processes and only adds API pseudo devices such as /dev/null, /dev/zero or /dev/random (as well as the pseudo TTY subsystem) to it, but no physical devices such as /dev/sda. This is useful to securely turn off physical device access by the executed process. ***** Defaults to false. ***** Enabling this option will also remove CAP_MKNOD from the capability bounding set for the unit (see above), and set DevicePolicy=closed (see systemd.resource-control(5) for details).Note that using this setting will disconnect propagation of mounts from the service to the host (propagation in the opposite direction continues to work). This means that this setting may not be used for services which shall be able to install mount points in the main mount namespace. The /dev namespace will be mounted read-only and 'noexec'. The latter may break old programs which try to set up executable memory by using mmap(2) of /dev/zero instead of using MAP_ANON. </snip> If I understood correctly, the default suppose to be false but currently systemd is shipping it as default true. Is that correct? > Also do you know what was the last version where this was working? We did > not change anything in that area recently. That's the weird part, the diff shows: <snip> -systemd-219-27.el7.src.rpm +systemd-219-30.el7.src.rpm </snip> Based on that I have rebuild the RHEV-H with systemd-219-27 and now I see this behaviour that in the past didn't show up. Anyway, here the full diff between the RHEV-H iso that works against the iso from today with the same systemd but still failing: --- rhev-hypervisor7-7.3-20160901.1.iso.d/isolinux/manifest-srpm.txt 2016-09-02 10:07:50.000000000 -0400 +++ rhev-hypervisor7-7.3-20160922.2.iso.d/isolinux/manifest-srpm.txt 2016-09-22 16:07:18.000000000 -0400 -NetworkManager-1.0.6-30.el7_2.src.rpm -OpenIPMI-2.0.19-11.el7.src.rpm +GeoIP-1.5.0-11.el7.src.rpm +NetworkManager-1.4.0-8.el7.src.rpm +OpenIPMI-2.0.19-15.el7.src.rpm -alsa-lib-1.0.28-2.el7.src.rpm +alsa-lib-1.1.1-1.el7.src.rpm -audit-2.4.1-5.el7.src.rpm +audit-2.6.5-3.el7.src.rpm -bind-9.9.4-29.el7_2.3.src.rpm +bind-9.9.4-36.el7.src.rpm -biosdevname-0.6.2-1.el7.src.rpm +biosdevname-0.7.2-1.el7.src.rpm -btrfs-progs-3.19.1-1.el7.src.rpm +btrfs-progs-4.4.1-1.el7.src.rpm -coreutils-8.22-15.el7_2.1.src.rpm +coreutils-8.22-18.el7.src.rpm -dbus-1.6.12-14.el7_2.src.rpm +dbus-1.6.12-17.el7.src.rpm -device-mapper-multipath-0.4.9-85.el7_2.5.src.rpm +device-mapper-multipath-0.4.9-99.el7.src.rpm -dhcp-4.2.5-42.el7.src.rpm +dhcp-4.2.5-47.el7.src.rpm -dmidecode-2.12-9.el7.src.rpm +dmidecode-3.0-2.el7.src.rpm -dnsmasq-2.66-14.el7_1.src.rpm +dnsmasq-2.66-21.el7.src.rpm -dracut-033-360.el7_2.1.src.rpm +dracut-033-462.el7.src.rpm -e2fsprogs-1.42.9-7.el7.src.rpm -ebtables-2.0.10-13.el7.src.rpm -efibootmgr-0.8.0-7.el7.src.rpm +e2fsprogs-1.42.9-9.el7.src.rpm +ebtables-2.0.10-15.el7.src.rpm +efibootmgr-0.8.0-10.el7.src.rpm -ethtool-3.15-2.el7.src.rpm +ethtool-4.5-3.el7.src.rpm -fcoe-utils-1.0.30-3.git91c0c8c.el7.src.rpm -fence-agents-4.0.11-27.el7_2.7.src.rpm -fence-virt-0.3.2-2.el7.src.rpm +fcoe-utils-1.0.31-1.git5dfd3e4.el7.src.rpm +fence-agents-4.0.11-47.el7.src.rpm +fence-virt-0.3.2-5.el7.src.rpm -firewalld-0.3.9-14.el7.src.rpm +firewalld-0.4.3.2-8.el7.src.rpm -fontconfig-2.10.95-7.el7.src.rpm +fontconfig-2.10.95-9.el7.src.rpm -freetype-2.4.11-11.el7.src.rpm -fuse-2.9.2-6.el7.src.rpm +freetype-2.4.11-12.el7.src.rpm +fuse-2.9.2-7.el7.src.rpm -ghostscript-9.07-18.el7.src.rpm +ghostscript-9.07-20.el7.src.rpm -glib2-2.42.2-5.el7.src.rpm +glib2-2.46.2-4.el7.src.rpm -glusterfs-3.7.1-16.el7.src.rpm +glusterfs-3.7.9-12.el7.src.rpm -gnutls-3.3.8-14.el7_2.src.rpm +gnutls-3.3.24-1.el7.src.rpm -grubby-8.28-17.el7.src.rpm +grubby-8.28-18.el7.src.rpm -gssproxy-0.4.1-8.el7_2.src.rpm +gssproxy-0.4.1-13.el7.src.rpm -gtk3-3.14.13-16.el7.src.rpm +gtk3-3.14.13-20.el7.src.rpm -hwdata-0.252-8.1.el7.src.rpm +hwdata-0.252-8.2.el7.src.rpm -initscripts-9.49.30-1.el7_2.3.src.rpm +initscripts-9.49.37-1.el7.src.rpm -ipmitool-1.8.13-8.el7_1.src.rpm -iproute-3.10.0-54.el7_2.1.src.rpm -iptables-1.4.21-16.el7.src.rpm -iputils-20121221-7.el7.src.rpm +ipmitool-1.8.15-7.el7.src.rpm +iproute-3.10.0-74.el7.src.rpm +ipset-6.19-6.el7.src.rpm +iptables-1.4.21-17.el7.src.rpm +iputils-20160308-8.el7.src.rpm -irqbalance-1.0.7-5.el7.src.rpm -iscsi-initiator-utils-6.2.0.873-33.el7_2.2.src.rpm +irqbalance-1.0.7-6.el7.src.rpm +iscsi-initiator-utils-6.2.0.873-35.el7.src.rpm +jansson-2.4-6.el7.src.rpm -kernel-3.10.0-500.el7.src.rpm -kexec-tools-2.0.7-38.el7_2.1.src.rpm +kernel-3.10.0-506.el7.src.rpm +kexec-tools-2.0.7-50.el7.src.rpm -kmod-20-5.el7.src.rpm -krb5-1.14.1-24.el7.src.rpm -lcms2-2.6-2.el7.src.rpm +kmod-20-9.el7.src.rpm +krb5-1.14.1-26.el7.src.rpm +lcms2-2.6-3.el7.src.rpm -libX11-1.6.3-2.el7.src.rpm +libX11-1.6.3-3.el7.src.rpm +libcacard-2.5.2-2.el7.src.rpm -libcgroup-0.41-8.el7.src.rpm +libcgroup-0.41-11.el7.src.rpm -libgovirt-0.3.3-1.el7_2.4.src.rpm +libgovirt-0.3.3-4.el7.src.rpm -libhbaapi-2.2.9-6.el7.src.rpm -libhbalinux-1.0.17-2.el7.src.rpm -libiscsi-1.9.0-6.el7.src.rpm +libiscsi-1.9.0-7.el7.src.rpm -libmlx4-1.0.6-5.el7.src.rpm +libmlx4-1.2.1-1.el7.src.rpm -libndp-1.2-6.el7_2.src.rpm +libndp-1.2-7.el7.src.rpm -libnfsidmap-0.25-12.el7.src.rpm +libnfsidmap-0.25-15.el7.src.rpm -libnl3-3.2.21-10.el7.src.rpm +libnl3-3.2.28-2.el7.src.rpm -libosinfo-0.2.12-3.el7.src.rpm +libosinfo-0.3.0-3.el7.src.rpm -librdmacm-1.0.21-1.el7.src.rpm +librdmacm-1.1.0-2.el7.src.rpm -libseccomp-2.2.1-1.el7.src.rpm +libseccomp-2.3.1-2.el7.src.rpm -libsoup-2.48.1-3.el7.src.rpm +libsoup-2.48.1-6.el7.src.rpm -libvirt-1.2.17-13.el7_2.5.src.rpm +libvirt-2.0.0-9.el7.src.rpm -libvirt-python-1.2.17-2.el7.src.rpm +libvirt-python-2.0.0-2.el7.src.rpm -linux-firmware-20160728-48.git0daeaf3.el7.src.rpm -lldpad-1.0.1-2.git986eb2e.el7.src.rpm -lm_sensors-3.3.4-11.el7.src.rpm -logrotate-3.8.6-7.el7_2.src.rpm +linux-firmware-20160830-49.git7534e19.el7.src.rpm +lldpad-1.0.1-3.git036e314.el7.src.rpm +lm_sensors-3.4.0-4.20160601gitf9185e5.el7.src.rpm +logrotate-3.8.6-12.el7.src.rpm -lsscsi-0.27-3.el7.src.rpm +lsscsi-0.27-4.el7.src.rpm -lua-5.1.4-14.el7.src.rpm -lvm2-2.02.130-5.el7_2.5.src.rpm +lua-5.1.4-15.el7.src.rpm +lvm2-2.02.165-2.el7.src.rpm -mariadb-5.5.50-1.el7_2.src.rpm -mcelog-120-3.e7e0ac1.el7.src.rpm -mdadm-3.3.2-7.el7_2.1.src.rpm +mariadb-5.5.50-2.el7.src.rpm +mcelog-136-1.e4aca63.el7.src.rpm +mdadm-3.4-12.el7.src.rpm -mom-0.5.5-1.el7ev.src.rpm +mom-0.5.6-1.el7ev.src.rpm -nfs-utils-1.3.0-0.21.el7_2.1.src.rpm +nfs-utils-1.3.0-0.33.el7.src.rpm -ntp-4.2.6p5-22.el7_2.2.src.rpm +ntp-4.2.6p5-25.el7.src.rpm -numad-0.5-14.20140620git.el7.src.rpm +numad-0.5-17.20150602git.el7.src.rpm -openssh-6.6.1p1-25.el7_2.src.rpm +openssh-6.6.1p1-31.el7.src.rpm -os-prober-1.58-5.el7.src.rpm +os-prober-1.58-8.el7.src.rpm -ovirt-node-3.6.1-15.0.el7ev.src.rpm +ovirt-node-3.6.1-18.0.el7ev.src.rpm -ovirt-node-plugin-vdsm-0.6.1-10.el7ev.src.rpm +ovirt-node-plugin-vdsm-0.6.1-11.el7ev.src.rpm -pam-1.1.8-12.el7_1.1.src.rpm +pam-1.1.8-18.el7.src.rpm -parted-3.1-23.el7.src.rpm +parted-3.1-28.el7.src.rpm -pciutils-3.2.1-4.el7.src.rpm +pciutils-3.5.1-1.el7.src.rpm -perl-5.16.3-286.el7.src.rpm +perl-5.16.3-291.el7.src.rpm -pixman-0.32.6-3.el7.src.rpm +pixman-0.34.0-1.el7.src.rpm -plymouth-0.8.9-0.24.20140113.el7.src.rpm +plymouth-0.8.9-0.26.20140113.el7.src.rpm -pulseaudio-6.0-7.el7.src.rpm +pulseaudio-6.0-8.el7.src.rpm -python-2.7.5-38.el7_2.src.rpm +python-2.7.5-48.el7.src.rpm -python-rhsm-1.15.4-5.el7.src.rpm +python-rhsm-1.17.9-1.el7.src.rpm -quota-4.01-11.el7_2.1.src.rpm +quota-4.01-14.el7.src.rpm -rdma-7.2_4.1_rc6-2.el7.src.rpm +rdma-7.3_4.7_rc2-5.el7.src.rpm -redhat-logos-70.0.3-4.el7.src.rpm -redhat-release-workstation-7.2-8.el7.src.rpm -rest-0.7.92-3.el7.src.rpm +redhat-logos-70.0.3-6.el7.src.rpm +redhat-release-client-7.3-3.el7.src.rpm +rest-0.7.92-5.el7.src.rpm -rhn-client-tools-2.0.2-6.el7.src.rpm +rhn-client-tools-2.0.2-8.el7.src.rpm -rpcbind-0.2.0-33.el7_2.1.src.rpm +rpcbind-0.2.0-38.el7.src.rpm -seabios-1.7.5-11.el7.src.rpm +seabios-1.9.1-4.el7.src.rpm -selinux-policy-3.13.1-96.el7.src.rpm +selinux-policy-3.13.1-99.el7.src.rpm -setup-2.8.71-6.el7.src.rpm +setup-2.8.71-7.el7.src.rpm -sos-3.2-35.el7_2.3.src.rpm -spice-0.12.4-18.el7.src.rpm -spice-gtk-0.26-5.el7.src.rpm +sos-3.3-4.el7.src.rpm +spice-0.12.4-19.el7.src.rpm +spice-gtk-0.31-6.el7.src.rpm +spice-protocol-0.12.11-1.el7.src.rpm -subscription-manager-1.15.9-15.el7.src.rpm +subscription-manager-1.17.15-1.el7.src.rpm -sysstat-10.1.5-7.el7.src.rpm +sysstat-10.1.5-11.el7.src.rpm -systemtap-2.8-10.el7.src.rpm +systemtap-3.0-6.el7.src.rpm -tar-1.26-30.el7.src.rpm +tar-1.26-31.el7.src.rpm -telnet-0.17-59.el7.src.rpm +telnet-0.17-60.el7.src.rpm -tuned-2.5.1-4.el7_2.3.src.rpm +tuned-2.7.1-3.el7.src.rpm -unbound-1.4.20-26.el7.src.rpm +unbound-1.4.20-28.el7.src.rpm -util-linux-2.23.2-26.el7_2.3.src.rpm -vdsm-4.17.34-1.el7ev.src.rpm +util-linux-2.23.2-33.el7.src.rpm +vdsm-4.17.35-1.el7ev.src.rpm -virt-viewer-2.0-6.el7_2.2.src.rpm -virt-what-1.13-6.el7.src.rpm -virt-who-0.14-9.el7_2.1.src.rpm -wget-1.14-10.el7_0.1.src.rpm +virt-viewer-2.0-12.el7.src.rpm +virt-what-1.13-8.el7.src.rpm +virt-who-0.17-10.el7.src.rpm +wget-1.14-13.el7.src.rpm -wpa_supplicant-2.0-17.el7_1.src.rpm -xfsprogs-4.5.0-6.el7.src.rpm +wpa_supplicant-2.0-20.el7.src.rpm +xfsprogs-4.5.0-8.el7.src.rpm -yum-3.4.3-132.el7.src.rpm +yum-3.4.3-149.el7.src.rpm -yum-rhn-plugin-2.0.1-5.el7.src.rpm -yum-utils-1.1.31-34.el7.src.rpm -zlib-1.2.7-15.el7.src.rpm +yum-rhn-plugin-2.0.1-6.el7.src.rpm +yum-utils-1.1.31-40.el7.src.rpm +zlib-1.2.7-17.el7.src.rpm Considering comment 22 I wonder if this is a Node only bug. If it's easy to reproduce on RHEL, then we need a systemd fix. But if we can't then we should use the workaround of comment 2. We should recognize that we had to add tehse kind of "fixes" for 7.0 and 7.1. *** This bug has been marked as a duplicate of bug 1381666 *** |
Description of problem: RHEV-H 7.3 boot is failing to start these services affecting the hypervisor TUI and virtual machine start. Version-Release number of selected component (if applicable): systemd-python-219-30.el7.x86_64 systemd-sysv-219-30.el7.x86_64 systemd-libs-219-30.el7.x86_64 systemd-219-30.el7.x86_64 dbus-glib-0.100-7.el7.x86_64 dbus-1.6.12-17.el7.x86_64 dbus-libs-1.6.12-17.el7.x86_64 dbus-python-1.1.1-9.el7.x86_64 kernel-3.10.0-506.el7.x86_64 How reproducible: - Install rhev-hypervisor7-7.3-20160921.1 - After auto-registration, storage added into RHEV-M the virtual machine fails to start.(systemd-machined) or - In the TUI Menu it will generate an exception when trying to configure keyboard (systemd-localed.service) Additional data: Selinux doesn't seem related # /bin/systemctl status -l dbus.service ● dbus.service - D-Bus System Message Bus Loaded: loaded (/usr/lib/systemd/system/dbus.service; static; vendor preset: disabled) Active: active (running) since Thu 2016-09-22 03:55:21 UTC; 42min ago Main PID: 1162 (dbus-daemon) CGroup: /system.slice/dbus.service └─1162 /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation Sep 22 03:55:21 localhost systemd[1]: Started D-Bus System Message Bus. Sep 22 03:55:22 localhost systemd[1]: Starting D-Bus System Message Bus... Sep 22 03:56:12 localhost dbus[1162]: [system] Activating via systemd: service name='org.freedesktop.locale1' unit='dbus-org.freedesktop.locale1.service' Sep 22 03:56:12 localhost dbus-daemon[1162]: dbus[1162]: [system] Activating via systemd: service name='org.freedesktop.locale1' unit='dbus-org.freedesktop.locale1.service' Sep 22 03:56:37 localhost dbus[1162]: [system] Failed to activate service 'org.freedesktop.locale1': timed out Sep 22 03:56:37 localhost dbus-daemon[1162]: dbus[1162]: [system] Failed to activate service 'org.freedesktop.locale1': timed out Sep 22 03:59:49 localhost dbus[1162]: [system] Activating via systemd: service name='org.freedesktop.locale1' unit='dbus-org.freedesktop.locale1.service' Sep 22 04:00:14 localhost dbus[1162]: [system] Failed to activate service 'org.freedesktop.locale1': timed out Sep 22 04:02:32 localhost dbus[1162]: [system] Activating via systemd: service name='org.freedesktop.machine1' unit='dbus-org.freedesktop.machine1.service' Sep 22 04:02:57 localhost dbus[1162]: [system] Failed to activate service 'org.freedesktop.machine1': timed out # /bin/systemctl status -l systemd-machined.service.service ● systemd-machined.service - Virtual Machine and Container Registration Service Loaded: loaded (/usr/lib/systemd/system/systemd-machined.service; static; vendor preset: disabled) Active: failed (Result: exit-code) since Thu 2016-09-22 04:02:32 UTC; 39min ago Docs: man:systemd-machined.service(8) http://www.freedesktop.org/wiki/Software/systemd/machined Process: 18317 ExecStart=/usr/lib/systemd/systemd-machined (code=exited, status=226/NAMESPACE) Main PID: 18317 (code=exited, status=226/NAMESPACE) Sep 22 04:02:32 localhost systemd[1]: Starting Virtual Machine and Container Registration Service... Sep 22 04:02:32 localhost systemd[1]: systemd-machined.service: main process exited, code=exited, status=226/NAMESPACE Sep 22 04:02:32 localhost systemd[1]: Failed to start Virtual Machine and Container Registration Service. Sep 22 04:02:32 localhost systemd[1]: Unit systemd-machined.service entered failed state. Sep 22 04:02:32 localhost systemd[1]: systemd-machined.service failed. Exception when trying to configure keyboard via TUI Traceback (most recent call last): File "<stdin>", line 2, in <module> File "/usr/lib/python2.7/site-packages/ovirtnode/ovirtfunctions.py", line 1726, in load_keyboard_config kbd = osystem.Keyboard() File "/usr/lib/python2.7/site-packages/ovirt/node/utils/system.py", line 726, in __init__ self.kbd = system_config_keyboard.keyboard.Keyboard() File "/usr/lib/python2.7/site-packages/system_config_keyboard/keyboard.py", line 60, in __init__ self._localed_wrap = localed.LocaledWrapper() File "/usr/lib/python2.7/site-packages/system_config_keyboard/localed.py", line 48, in __init__ raise LocaledWrapperError("Failed to get locale object") system_config_keyboard.localed.LocaledWrapperError: Failed to get locale object /bin/systemctl status -l systemd-localed.service ● systemd-localed.service - Locale Service Loaded: loaded (/usr/lib/systemd/system/systemd-localed.service; static; vendor preset: disabled) Active: failed (Result: exit-code) since Thu 2016-09-22 04:40:22 UTC; 715ms ago Docs: man:systemd-localed.service(8) man:locale.conf(5) man:vconsole.conf(5) http://www.freedesktop.org/wiki/Software/systemd/localed Process: 21147 ExecStart=/usr/lib/systemd/systemd-localed (code=exited, status=226/NAMESPACE) Main PID: 21147 (code=exited, status=226/NAMESPACE) Sep 22 04:40:22 localhost systemd[1]: Starting Locale Service... Sep 22 04:40:22 localhost systemd[1]: systemd-localed.service: main process exited, code=exited, status=226/NAMESPACE Sep 22 04:40:22 localhost systemd[1]: Failed to start Locale Service. Sep 22 04:40:22 localhost systemd[1]: Unit systemd-localed.service entered failed state. Sep 22 04:40:22 localhost systemd[1]: systemd-localed.service failed. # ldd /usr/lib/systemd/systemd-localed linux-vdso.so.1 => (0x00007ffd3ad24000) libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fef783a4000) librt.so.1 => /lib64/librt.so.1 (0x00007fef7819c000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fef77f85000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fef77d69000) libc.so.6 => /lib64/libc.so.6 (0x00007fef779a8000) /lib64/ld-linux-x86-64.so.2 (0x00007fef7862c000) libpcre.so.1 => /lib64/libpcre.so.1 (0x00007fef77746000) libdl.so.2 => /lib64/libdl.so.2 (0x00007fef77542000) # cat /lib/systemd/system/systemd-localed.service # This file is part of systemd. # # systemd is free software; you can redistribute it and/or modify it # under the terms of the GNU Lesser General Public License as published by # the Free Software Foundation; either version 2.1 of the License, or # (at your option) any later version. [Unit] Description=Locale Service Documentation=man:systemd-localed.service(8) man:locale.conf(5) man:vconsole.conf(5) Documentation=http://www.freedesktop.org/wiki/Software/systemd/localed [Service] ExecStart=/usr/lib/systemd/systemd-localed BusName=org.freedesktop.locale1 CapabilityBoundingSet= WatchdogSec=3min PrivateTmp=yes PrivateDevices=yes PrivateNetwork=yes ProtectSystem=yes ProtectHome=yes