Hide Forgot
Description of problem: Systemd shows different errors while running in a docker container unprivileged, the application tested is ManageIQ containerized. Version-Release number of selected component (if applicable): systemd-229-16.fc24.x86_64 docker-1.10.3-54.gite03ddb8.fc24.x86_64 oci-systemd-hook-0.1.4-1.fc24.x86_64 kernel-4.8.6-201.fc24.x86_64 oci-register-machine-0-2.4.git352a2a2.fc24.x86_64 How reproducible: Always Steps to Reproduce: 1. docker run -d -p 80:80 -p 443:443 manageiq/manageiq 2. machinectl to obtain <machine-id> 3. journalctl -M <machine-id> Actual results: Nov 16 18:37:29 d8b377779dc1 systemd-journal[20]: Runtime journal is using 4.0M (max allowed 8.0M, trying to leave 9.6M free of 59.9M available → current limit 8.0M). Nov 16 18:37:29 d8b377779dc1 systemd-journal[20]: Permanent journal is using 8.0M (max allowed 1022.2M, trying to leave 1.4G free of 8.3G available → current limit 1022.2M). Nov 16 18:37:29 d8b377779dc1 systemd-journal[20]: Time spent on flushing to /var is 783us for 2 entries. Nov 16 18:37:29 d8b377779dc1 systemd-journal[20]: Journal started Nov 16 18:37:29 d8b377779dc1 systemd[1]: Started Create Volatile Files and Directories. Nov 16 18:37:29 d8b377779dc1 systemd[1]: Dependency failed for Update UTMP about System Runlevel Changes. Nov 16 18:37:29 d8b377779dc1 systemd[1]: Job systemd-update-utmp-runlevel.service/start failed with result 'dependency'. Nov 16 18:37:29 d8b377779dc1 systemd[1]: Reached target System Initialization. Nov 16 18:37:29 d8b377779dc1 systemd[1]: Starting System Initialization. Nov 16 18:37:29 d8b377779dc1 systemd[1]: Started Daily Cleanup of Temporary Directories. Nov 16 18:37:29 d8b377779dc1 systemd[1]: Starting Daily Cleanup of Temporary Directories. Nov 16 18:37:29 d8b377779dc1 systemd[1]: Reached target Timers. Nov 16 18:37:29 d8b377779dc1 systemd[1]: Starting Timers. Nov 16 18:37:29 d8b377779dc1 systemd[1]: Listening on D-Bus System Message Bus Socket. Nov 16 18:37:29 d8b377779dc1 systemd[1]: Starting D-Bus System Message Bus Socket. Nov 16 18:37:29 d8b377779dc1 systemd[1]: Reached target Sockets. Nov 16 18:37:29 d8b377779dc1 systemd[1]: Starting Sockets. Nov 16 18:37:29 d8b377779dc1 systemd[1]: Reached target Basic System. Nov 16 18:37:29 d8b377779dc1 systemd[1]: Starting Basic System. Nov 16 18:37:29 d8b377779dc1 systemd[1]: Starting "Initializes the evm environment"... Nov 16 18:37:29 d8b377779dc1 systemd[1]: Starting LSB: Bring up/down networking... Nov 16 18:37:29 d8b377779dc1 systemd[1]: Starting "top(1) data collection"... Nov 16 18:37:29 d8b377779dc1 systemd[1]: Started Memcached. Nov 16 18:37:29 d8b377779dc1 systemd[1]: Starting Memcached... Nov 16 18:37:29 d8b377779dc1 systemd[1]: Starting "vmstat(1) data collection"... Nov 16 18:37:29 d8b377779dc1 systemd[1]: Starting Cleanup of Temporary Directories... Nov 16 18:37:29 d8b377779dc1 systemd[1]: Started Cleanup of Temporary Directories. Nov 16 18:37:29 d8b377779dc1 systemd[1]: Started "top(1) data collection". Nov 16 18:37:29 d8b377779dc1 systemd[1]: Started "vmstat(1) data collection". Nov 16 18:37:29 d8b377779dc1 systemd-sysctl[69]: Failed to write '0' to '/proc/sys/net/ipv4/conf/all/accept_source_route': Read-only file system Nov 16 18:37:29 d8b377779dc1 systemd-sysctl[69]: Failed to write '16' to '/proc/sys/kernel/sysrq': Read-only file system Nov 16 18:37:29 d8b377779dc1 systemd-sysctl[69]: Failed to write '1' to '/proc/sys/net/ipv4/conf/all/promote_secondaries': Read-only file system Nov 16 18:37:29 d8b377779dc1 systemd-sysctl[69]: Failed to write '0' to '/proc/sys/net/ipv4/conf/default/accept_source_route': Read-only file system Nov 16 18:37:29 d8b377779dc1 systemd-sysctl[69]: Failed to write '1' to '/proc/sys/net/ipv4/conf/default/rp_filter': Read-only file system Nov 16 18:37:29 d8b377779dc1 systemd-sysctl[69]: Failed to write '1' to '/proc/sys/fs/protected_hardlinks': Read-only file system Nov 16 18:37:29 d8b377779dc1 systemd-sysctl[69]: Failed to write '1' to '/proc/sys/net/ipv4/conf/all/rp_filter': Read-only file system Nov 16 18:37:29 d8b377779dc1 systemd-sysctl[69]: Failed to write '1' to '/proc/sys/net/ipv4/conf/default/promote_secondaries': Read-only file system Nov 16 18:37:29 d8b377779dc1 systemd-sysctl[69]: Failed to write '1' to '/proc/sys/kernel/core_uses_pid': Read-only file system Nov 16 18:37:29 d8b377779dc1 systemd-sysctl[69]: Failed to write '1' to '/proc/sys/fs/protected_symlinks': Read-only file system Nov 16 18:37:30 d8b377779dc1 sh[21]: /run/lvm/lvmetad.socket: connect failed: No such file or directory Nov 16 18:37:30 d8b377779dc1 sh[21]: WARNING: Failed to connect to lvmetad. Falling back to internal scanning. Nov 16 18:37:30 d8b377779dc1 sh[21]: No matching physical volumes found Nov 16 18:37:30 d8b377779dc1 systemd[1]: Started "Initializes the evm environment". Nov 16 18:37:30 d8b377779dc1 systemd[1]: Starting Initialize Appliance Database... Nov 16 18:37:30 d8b377779dc1 network[22]: Bringing up loopback interface: RTNETLINK answers: Operation not permitted Nov 16 18:37:30 d8b377779dc1 network[22]: ERROR : [/etc/sysconfig/network-scripts/ifup-eth] Failed to bring up lo. Nov 16 18:37:30 d8b377779dc1 /etc/sysconfig/network-scripts/ifup-eth[101]: Failed to bring up lo. Nov 16 18:37:30 d8b377779dc1 network[22]: [FAILED] Nov 16 18:37:30 d8b377779dc1 network[22]: RTNETLINK answers: Operation not permitted Nov 16 18:37:30 d8b377779dc1 network[22]: RTNETLINK answers: Operation not permitted Nov 16 18:37:30 d8b377779dc1 network[22]: RTNETLINK answers: Operation not permitted Nov 16 18:37:30 d8b377779dc1 network[22]: RTNETLINK answers: Operation not permitted Nov 16 18:37:30 d8b377779dc1 network[22]: RTNETLINK answers: Operation not permitted Nov 16 18:37:30 d8b377779dc1 network[22]: RTNETLINK answers: Operation not permitted Nov 16 18:37:30 d8b377779dc1 network[22]: RTNETLINK answers: Operation not permitted Nov 16 18:37:30 d8b377779dc1 network[22]: RTNETLINK answers: Operation not permitted Nov 16 18:37:30 d8b377779dc1 network[22]: RTNETLINK answers: Operation not permitted Nov 16 18:37:46 d8b377779dc1 systemd-logind[147]: New seat seat0. Nov 16 18:37:46 d8b377779dc1 systemd[1]: Created slice user-26.slice. Nov 16 18:37:46 d8b377779dc1 systemd[1]: Starting user-26.slice. Nov 16 18:37:46 d8b377779dc1 systemd[1]: Started Session c1 of user postgres. Nov 16 18:37:46 d8b377779dc1 systemd[1]: Starting Session c1 of user postgres. Nov 16 18:37:46 d8b377779dc1 runuser[145]: pam_unix(runuser-l:session): session opened for user postgres by (uid=0) Nov 16 18:37:46 d8b377779dc1 runuser[145]: pam_unix(runuser-l:session): session closed for user postgres Nov 16 18:37:46 d8b377779dc1 systemd[1]: Removed slice user-26.slice. Nov 16 18:37:46 d8b377779dc1 systemd[1]: Stopping user-26.slice. Nov 16 18:37:46 d8b377779dc1 systemd-logind[147]: Failed to unmount user runtime directory /run/user/26: Operation not permitted Nov 16 18:37:46 d8b377779dc1 systemd[1]: Created slice user-26.slice. Nov 16 18:37:46 d8b377779dc1 systemd[1]: Starting user-26.slice. Nov 16 18:37:46 d8b377779dc1 systemd[1]: Started Session c2 of user postgres. Nov 16 18:37:46 d8b377779dc1 systemd[1]: Starting Session c2 of user postgres. Nov 16 18:37:46 d8b377779dc1 runuser[164]: pam_unix(runuser-l:session): session opened for user postgres by (uid=0) Nov 16 18:37:54 d8b377779dc1 runuser[164]: pam_unix(runuser-l:session): session closed for user postgres Nov 16 18:37:54 d8b377779dc1 systemd[1]: Removed slice user-26.slice. Nov 16 18:37:54 d8b377779dc1 systemd[1]: Stopping user-26.slice. Nov 16 18:37:54 d8b377779dc1 systemd-logind[147]: Failed to unmount user runtime directory /run/user/26: Operation not permitted Nov 16 18:37:54 d8b377779dc1 systemd[1]: Reloading. Expected results: No errors while running unprivileged systemd containers. Additional info: These errors do not cause the container to crash but they seem troubling.
Should we remove the contents of /lib/sysctl.d from the base container so that these errors do not happen in a systemd based container? Nov 16 18:37:29 d8b377779dc1 systemd-sysctl[69]: Failed to write '16' to '/proc/sys/kernel/sysrq': Read-only file system cd /lib/sysctl.d/ # ls 00-system.conf 50-coredump.conf 60-libvirtd.conf 10-default-yama-scope.conf 50-default.conf # rpm -qf * initscripts-9.69-1.fc26.x86_64 elfutils-default-yama-scope-0.167-2.fc26.noarch systemd-231-11.fc26.x86_64 systemd-231-11.fc26.x86_64 libvirt-daemon-2.4.0-1.fc26.x86_64
How do we prevent the networks from attempting to come up?
This bug appears to have been reported against 'rawhide' during the Fedora 27 development cycle. Changing version to '27'.
This bug has now also been introduced on CentOS 7.5 running in an unprivileged lxc container after updating filesystem to filesystem-3.2-25.el7.x86_64. May 12 13:48:23 sdn systemd-sysctl: Failed to write '0' to '/proc/sys/kernel/yama/ptrace_scope': Read-only file system May 12 13:48:23 sdn systemd-sysctl: Failed to write '16' to '/proc/sys/kernel/sysrq': Read-only file system May 12 13:48:23 sdn systemd-sysctl: Failed to write '1' to '/proc/sys/kernel/core_uses_pid': Read-only file system May 12 13:48:23 sdn systemd-sysctl: Failed to write '1' to '/proc/sys/fs/protected_hardlinks': Read-only file system May 12 13:48:23 sdn systemd-sysctl: Failed to write '1' to '/proc/sys/fs/protected_symlinks': Read-only file system May 12 13:48:31 sdn systemd-sysctl: Failed to write '0' to '/proc/sys/kernel/yama/ptrace_scope': Read-only file system May 12 13:48:31 sdn systemd-sysctl: Failed to write '16' to '/proc/sys/kernel/sysrq': Read-only file system May 12 13:48:31 sdn systemd-sysctl: Failed to write '1' to '/proc/sys/kernel/core_uses_pid': Read-only file system May 12 13:48:31 sdn systemd-sysctl: Failed to write '1' to '/proc/sys/fs/protected_hardlinks': Read-only file system May 12 13:48:31 sdn systemd-sysctl: Failed to write '1' to '/proc/sys/fs/protected_symlinks': Read-only file system
...additionally, this one comes up after the update, which I find rather troubling: May 12 13:48:23 sdn systemd-remount-fs: /bin/mount for / exited with exit status 32. May 12 13:48:23 sdn mount: mount: permission denied May 12 13:48:23 sdn systemd: Starting Flush Journal to Persistent Storage... May 12 13:48:23 sdn mount: mount: permission denied May 12 13:48:23 sdn mount: mount: configfs is write-protected, mounting read-only May 12 13:48:23 sdn mount: mount: cannot mount configfs read-only May 12 13:48:23 sdn systemd-remount-fs: mount: cannot remount /dev/root read-write, is write-protected
Hm, downgrading to filesystem-3.2-21.el7.x86_64 did not make any change to this behaviour, so maybe the filesystem update is unrelated and this was a coincidence. Removing the contents from /lib/sysctl.d/ does indeed remove all the Read-only file system warnings, still leaving me with the remount errors above. As it turns out, I am also getting these errors when the container is running privileged. # mount zfs/subvol-108-disk-1 on / type zfs (rw,xattr,posixacl) none on /dev type tmpfs (rw,relatime,size=492k,mode=755) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) proc on /proc/sys/net type proc (rw,nosuid,nodev,noexec,relatime) proc on /proc/sys type proc (ro,nosuid,nodev,noexec,relatime) proc on /proc/sysrq-trigger type proc (ro,nosuid,nodev,noexec,relatime) sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime) sysfs on /sys/devices/virtual/net type sysfs (rw,relatime) sysfs on /sys/devices/virtual/net type sysfs (rw,nosuid,nodev,noexec,relatime) lxcfs on /proc/cpuinfo type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other) lxcfs on /proc/diskstats type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other) lxcfs on /proc/meminfo type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other) lxcfs on /proc/stat type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other) lxcfs on /proc/swaps type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other) lxcfs on /proc/uptime type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other) fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime) udev on /dev/net type devtmpfs (rw,nosuid,relatime,size=16402888k,nr_inodes=4100722,mode=755) devpts on /dev/lxc/console type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) devpts on /dev/console type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666,max=1024) devpts on /dev/ptmx type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666,max=1024) devpts on /dev/lxc/tty1 type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666,max=1024) devpts on /dev/lxc/tty2 type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666,max=1024) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) mqueue on /dev/mqueue type mqueue (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=52432k,mode=700) # cat /etc/fstab /dev/root / rootfs defaults 0 0
Hm, I just verified the behaviour with a CentOS 7.4 base lxc image and I am seeing already the exact same problems, just that logging looked different and that's why I was never alerted by monitoring about this. I apologize, I need to correct myself: a) The problem did not recently get introduced, it was already present in at least CentOS 7.4 (Dec 2017 image). b) filesystem-3.2-25.el7.x86_64 does not have anything to do with this.
This message is a reminder that Fedora 27 is nearing its end of life. On 2018-Nov-30 Fedora will stop maintaining and issuing updates for Fedora 27. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '27'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 27 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
This message is a reminder that Fedora 29 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora 29 on 2019-11-26. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '29'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 29 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Fedora 29 changed to end-of-life (EOL) status on 2019-11-26. Fedora 29 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.