Bug 892927 - "ip netns exec" unmounts /sys/fs/cgroup
Summary: "ip netns exec" unmounts /sys/fs/cgroup
Keywords:
Status: CLOSED DUPLICATE of bug 882047
Alias: None
Product: Fedora
Classification: Fedora
Component: iproute
Version: 18
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
Assignee: Petr Šabata
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-01-08 09:27 UTC by Matthias Runge
Modified: 2013-01-10 15:02 UTC (History)
13 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2013-01-10 12:51:01 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
output of strace -o ... systemctl (9.75 KB, text/plain)
2013-01-09 07:21 UTC, Matthias Runge
no flags Details

Description Matthias Runge 2013-01-08 09:27:21 UTC
Description of problem:
when doing systemctl ...., I'm getting the following message:
Failed to get D-Bus connection: No connection to service manager.

[root@turing ~]# ps ax | grep dbus
  595 ?        Ss     0:04 /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
 3914 ?        S      0:00 dbus-launch --autolaunch d29598f4bbd517b59d8a20a90000000e --binary-syntax --close-stderr
 3915 ?        Ss     0:00 /bin/dbus-daemon --fork --print-pid 5 --print-address 7 --session
 3921 ?        S      0:00 /bin/dbus-daemon --config-file=/etc/at-spi2/accessibility.conf --nofork --print-address 3
 3957 ?        S      0:00 dbus-launch --sh-syntax --exit-with-session
 3958 ?        Ss     0:03 /bin/dbus-daemon --fork --print-pid 5 --print-address 7 --session
 4238 ?        S      0:00 /bin/dbus-daemon --config-file=/etc/at-spi2/accessibility.conf --nofork --print-address 3
23913 pts/6    S+     0:00 grep --color=auto dbus


dbus seems to run, so what's the issue here? I also disabled selinux.



Version-Release number of selected component (if applicable):
[root@turing ~]# rpm -qf /usr/bin/systemctl
systemd-195-15.fc18.x86_64


How reproducible:
right after system boot, systemctl works, but not after hours and/or system suspend/resume


Is there a workaround? The system is pretty useless this way, because one can not eg. reboot.

Comment 1 Michal Schmidt 2013-01-08 13:32:55 UTC
(In reply to comment #0)
> Description of problem:
> when doing systemctl ...., I'm getting the following message:
> Failed to get D-Bus connection: No connection to service manager.

"No connection ..." is a generic error message printed if we have no specific error from libdbus. If the DBus connection were attempted, I'd expect there to be a more specific error.

Could you run it under strace and attach the resulting file here?:
strace -o /root/systemctl.txt systemctl

Comment 2 Matthias Runge 2013-01-09 07:21:16 UTC
Created attachment 675289 [details]
output of strace -o ... systemctl

Comment 3 Matthias Runge 2013-01-09 07:23:09 UTC
This error seems somehow connected with suspend/resume.

Comment 4 Matthias Runge 2013-01-09 09:49:31 UTC
one other thing might trigger this:

Jan  9 09:47:57 turing dbus-daemon[623]: dbus[623]: [system] Rejected send message, 2 matched rules; type="method_return", sender=":1.0" (uid=0 pid=614 comm="/usr/lib/systemd/systemd-logind ") interface="(unset)" member="(unset)" error name="(unset)" requested_reply="0" destination=":1.46" (uid=1000 pid=2304 comm="/usr/bin/gnome-session --session=cinnamon ")
Jan  9 09:47:57 turing dbus[623]: [system] Rejected send message, 2 matched rules; type="method_return", sender=":1.0" (uid=0 pid=614 comm="/usr/lib/systemd/systemd-logind ") interface="(unset)" member="(unset)" error name="(unset)" requested_reply="0" destination=":1.46" (uid=1000 pid=2304 comm="/usr/bin/gnome-session --session=cinnamon ")

But it's probably unrelated, because I have seen this using XFCE, too.

Comment 5 Michal Schmidt 2013-01-09 13:56:27 UTC
From the strace:

lstat("/sys/fs/cgroup/systemd", 0x7ffffc06b940) = -1 ENOENT (No such file or directory)

This is bad. The systemd cgroup hierarchy must not disappear. Did it get unmounted? Or maybe /sys/fs/cgroup got over-mounted? Please check the output of "findmnt".

Comment 6 Matthias Runge 2013-01-10 07:38:02 UTC
Isn't that mount /sys/fs/cgroup managed by systemd?


[root@turing ~]# findmnt 
TARGET                         SOURCE             FSTYPE   OPTIONS
/                              /dev/sda2          ext4     rw,relatime,data=orde
├─/proc                        proc               proc     rw,nosuid,nodev,noexe
│ └─/proc/sys/fs/binfmt_misc   systemd-1          autofs   rw,relatime,fd=31,pgr
│   └─/proc/sys/fs/binfmt_misc binfmt_misc        binfmt_m rw,relatime
├─/dev                         devtmpfs           devtmpfs rw,nosuid,size=401727
│ ├─/dev/mqueue                mqueue             mqueue   rw,relatime
│ ├─/dev/shm                   tmpfs              tmpfs    rw,nosuid,nodev
│ ├─/dev/pts                   devpts             devpts   rw,nosuid,noexec,rela
│ └─/dev/hugepages             hugetlbfs          hugetlbf rw,relatime
├─/run                         tmpfs              tmpfs    rw,nosuid,nodev,mode=
│ ├─/run/netns/qdhcp-c8a47d83-bd18-4899-81a7-fe390fc9fe3f
                               proc[/4037/ns/net] proc     rw,nosuid,nodev,noexe
│ ├─/run/user/1000/gvfs        gvfsd-fuse         fuse.gvf rw,nosuid,nodev,relat
│ └─/run/netns/qdhcp-706c2ef4-6c71-43e3-a4be-0e666729f0d7
                               proc[/3980/ns/net] proc     rw,nosuid,nodev,noexe
├─/tmp                         tmpfs              tmpfs    rw
├─/boot                        /dev/sda1          ext4     rw,relatime,data=orde
└─/sys                         qdhcp-c8a47d83-bd18-4899-81a7-fe390fc9fe3f
                                                  sysfs    rw,relatime

This interesting situation happens after having the machine running for about 1hr (or so). I can't see a reason, why it should become unmounted. I don't use automount and also no other services to (un-) mount automatically.

Comment 7 Michal Schmidt 2013-01-10 09:50:20 UTC
(In reply to comment #6)
> Isn't that mount /sys/fs/cgroup managed by systemd?

It's simply mounted by systemd during early boot. It is not "managed" during normal system operation.

> │ ├─/run/netns/qdhcp-c8a47d83-bd18-4899-81a7-fe390fc9fe3f
...
> │ └─/run/netns/qdhcp-706c2ef4-6c71-43e3-a4be-0e666729f0d7

Any ideas what these mountpoints are?

> └─/sys                         qdhcp-c8a47d83-bd18-4899-81a7-fe390fc9fe3f

The second field is very unusual for the /sys mountpoint.

[after some googling...]

"qdhcp" appears in a couple of slides about "Quantum" and "OpenStack". Does it sound like something that you run on your machine?

Comment 8 Matthias Runge 2013-01-10 09:59:55 UTC
Yes, I have the full stack running here.

I disabled it for now, to see, how it turns out.

Even though there is the other mount point, I can't see, why that dir get's unmounted. Is there a workaround? Especially, I'm sure, OpenStack Quantum does not unmount /sys/fs/cgroup/

Comment 9 Michal Schmidt 2013-01-10 11:13:02 UTC
openstack-quantum sets up network namespaces by calling "ip netns exec ...".
The bug can be reproduced by:

ip netns add test
ip netns exec test echo hello

Afterwards, /sys is remounted anew, the groups are gone.

Comment 10 Petr Šabata 2013-01-10 12:51:01 UTC

*** This bug has been marked as a duplicate of bug 882047 ***

Comment 11 Matthias Runge 2013-01-10 13:32:31 UTC
Once that ip is currently identified as bad boy, I wonder why there couldn't be also a workaround implemented in systemd, because also something else might unmount/remount /sys, even an admin by accident. If done so, you can not reboot your system anymore.
So implementing a workaround in systemd, too might be seen as good will.

Comment 12 Petr Šabata 2013-01-10 14:05:53 UTC
I also wonder why the sysfs remount affects the original filesystem and mounts when it's executed after an unshare(CLONE_NEWNS) call...

Comment 13 Michal Schmidt 2013-01-10 14:36:30 UTC
> If done so, you can not reboot your system anymore.

When nothing else works, there's always "sync && reboot -f".

> So implementing a workaround in systemd, too might be seen as good will.

It's not obvious to me what the workaround should look like.

(In reply to comment #12)
> I also wonder why the sysfs remount affects the original filesystem and
> mounts when it's executed after an unshare(CLONE_NEWNS) call...

Because it's a shared mount. See:
http://www.mjmwired.net/kernel/Documentation/filesystems/sharedsubtree.txt

Comment 14 Petr Šabata 2013-01-10 15:00:08 UTC
(In reply to comment #13)
> (In reply to comment #12)
> > I also wonder why the sysfs remount affects the original filesystem and
> > mounts when it's executed after an unshare(CLONE_NEWNS) call...
> 
> Because it's a shared mount. See:
> http://www.mjmwired.net/kernel/Documentation/filesystems/sharedsubtree.txt

Shouldn't exactly that be solved by this call, making a private copy of it?

Comment 15 Michal Schmidt 2013-01-10 15:02:58 UTC
227	5) Detailed semantics:
228	-------------------
229		The section below explains the detailed semantics of
230		bind, rbind, move, mount, umount and clone-namespace operations.
...
572	5g) Clone Namespace
573	
574		A cloned namespace contains all the mounts as that of the parent
575		namespace.
576	
577		Let's say 'A' and 'B' are the corresponding mounts in the parent and the
578		child namespace.
579	
580		If 'A' is shared, then 'B' is also shared and 'A' and 'B' propagate to
581		each other.


Note You need to log in before you can comment on or make changes to this bug.