Bug 1213385 - Docker 1.6 mounts /var/log/journal/4809557f6c5a83f899a25b507d72c4d5
Summary: Docker 1.6 mounts /var/log/journal/4809557f6c5a83f899a25b507d72c4d5
Alias: None
Product: Fedora
Classification: Fedora
Component: docker-io
Version: 21
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
Assignee: Daniel Walsh
QA Contact: Fedora Extras Quality Assurance
Depends On:
Blocks: 1221482
TreeView+ depends on / blocked
Reported: 2015-04-20 12:57 UTC by Jan Pazdziora
Modified: 2015-05-14 07:09 UTC (History)
19 users (show)

Fixed In Version: docker-io-1.6.0-2.git3eac457.fc21
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1221482 (view as bug list)
Last Closed: 2015-04-26 12:53:52 UTC
Type: Bug

Attachments (Terms of Use)

Description Jan Pazdziora 2015-04-20 12:57:38 UTC
Description of problem:

With docker-io upgraded to 1.6, docker build and docker runs of https://github.com/adelton/docker-freeipa started to fail.

The reason seems to be that the container gets /var/log/journal/{id-of-the-container?} mounted and you cannot get rid of it because

[root@4809557f6c5a /]# umount -f /var/log/journal/4809557f6c5a83f899a25b507d72c4d5
umount: /var/log/journal/4809557f6c5a83f899a25b507d72c4d5: must be superuser to unmount

That however prevents use from moving /var/log to VOLUME because in build time, /var/log/canot be removed:

rm: cannot remove '/var/log/journal/cf00ac62f9dad464306c48f0267be479': Device or resource busy

Version-Release number of selected component (if applicable):

# rpm -q docker-io docker
package docker is not installed

How reproducible:


Steps to Reproduce:
1. Install docker-io.
2. Make no changes to /etc/sysconfig/docker except adding --icc=false to OPTIONS.
3. Run the service.
4. Run docker run -ti fedora:20 mount | grep /var/log

Actual results:

/dev/mapper/fedora--server_mgmt12-root on /var/log/journal/4809557f6c5a83f899a25b507d72c4d5 type ext4 (rw,relatime,seclabel,data=ordered)

Expected results:

Nothing mounted on /var/log.

Additional info:

Comment 2 Jan Pazdziora 2015-04-20 13:01:47 UTC
I see the same behaviour with docker-1.6.0-1.git0591dce.fc22.x86_64 but on Fedora 21 it's obviously more severe because it would mean regression there.

Comment 3 Lokesh Mandvekar 2015-04-20 18:34:27 UTC
Dan, I don't quite remember, you mentioned something about journald work sometime back, is that something not in fedora yet?

Comment 4 Daniel Walsh 2015-04-20 18:43:46 UTC
Jan What exactly are you trying to do, can you ping me on IRC to tell me exactly what is going on?

Comment 5 Daniel Walsh 2015-04-20 18:55:44 UTC
Lokesh this should work in Fedora but not RHEL7 yet.

Comment 6 Andy Goldstein 2015-04-20 19:06:32 UTC
From https://github.com/adelton/docker-freeipa/blob/master/Dockerfile

Step 14 : RUN cd / ; mkdir /data-template ; cat /etc/volume-data-list | while read i ; do if [ -e $i ] ; then tar cf - .$i | ( cd /data-template && tar xf - ) ; fi ; mkdir -p $( dirname $i ) ; rm -rf $i ; ln -sf /data${i%/} ${i%/} ; done
 ---> Running in 8ad03310a401
rm: cannot remove '/var/log/journal/8ad03310a40178c266b7b70bddbfb778': Device or resource busy

/etc/volume-data-list includes /var/log/

Comment 7 Daniel Walsh 2015-04-20 19:13:50 UTC
Ok you are just getting an error trying to remove the contents of the directory, but everything else should work fine.

I can see this being a potential issue if you are saving the logs in the image.

Comment 8 Jan Pazdziora 2015-04-21 06:50:01 UTC
In the FreeIPA image, I move /var/log to a /data volume and replace it with a symlink /var/log -> /data/var/log in the image. The goal is not to save logs in the image but into the volume.

Comment 9 Jan Pazdziora 2015-04-21 11:25:09 UTC
I could certainly do something like

RUN ... process list of directories to put to volume (== move aside to /data-template during build) ... if [ "$i" == /var/log/ ] ; then mv /var/log /var/log-removed ; else rm -rf $i ; fi ...
RUN rm -rf /var/log-removed

But why mount it directly to /var/log/journal/... in the first place? Why not something like /run/journal/data and have the /var/log/journal/... be a symlink to it? I assume for many people, having /var/log in a volume is and will be way to go, and preparing for journald support shouldn't break that.

I'm scared what other not-so-obvious features might be included in the future.

Comment 10 Jan Pazdziora 2015-04-21 11:26:17 UTC
(In reply to Daniel Walsh from comment #7)
> Ok you are just getting an error trying to remove the contents of the
> directory, but everything else should work fine.

If I can't remove /var/log/journal/8ad03310a40178c266b7b70bddbfb778, I can't remove /var/log, so I can't replace it with symlink pointing to /data.

Comment 11 Daniel Walsh 2015-04-21 13:43:40 UTC
Why not directly volume mount /var/log into the container?

-v /data/var/log:/var/log

Comment 12 Jan Pazdziora 2015-04-21 13:58:48 UTC
(In reply to Daniel Walsh from comment #11)
> Why not directly volume mount /var/log into the container?
> -v /data/var/log:/var/log

That'd be one more -v option to give to docker run.

The goal of FreeIPA is to integrate multiple things under one umbrella and hide the complexity of the various setups. We try hard to consolidate it all (via symlinks) into one volume mountpoint, avoiding the need for the admin to have numerous volumes and numerous -v options.

Besides, /var/log has rpm-populated directories that the software expects to find there. If they get hidden with an empty mounted volume, we are likely to see additional breakage.

Comment 13 Daniel Walsh 2015-04-21 14:23:39 UTC
Jan lets talk in IRC.

Comment 14 Andy Goldstein 2015-04-22 20:51:22 UTC
Any more updates on this?

Comment 15 Daniel Walsh 2015-04-22 21:15:07 UTC
Jan and I had a discussion on the atomic command and how to seup the DATADIR for volume mounting into the container.  I believe that this should be handled via a different mechanism then he is currently building.

Comment 16 Andy Goldstein 2015-04-22 21:32:16 UTC
Ok, great, thanks.

Jan, are you willing to close this, or is further resolution needed?

Comment 17 Jan Pazdziora 2015-04-23 16:59:53 UTC
(In reply to Andy Goldstein from comment #16)
> Jan, are you willing to close this, or is further resolution needed?

I'd certainly prefer if docker (the daemon, I assume) was not mounting additional external objects without the user asking for it, especially when they then cannot be unmounted. We've had enough problems being stuck with /etc/resolv.conf that couldn't be removed or unmounted, that got resolved and now new path pop up.

It certainly is not a blocker for the 1.6 release but I hope there is a better way to offer the additional functionality without forcing it on everyone unconditionally.

Comment 18 Daniel Walsh 2015-04-23 17:52:38 UTC
I would guess we could easily remove it from docker build, since there is little reason to do it there.  The question would be how do you get this type of functionality by default without adding mount points.

We also looking to add /run and /tmp as a tmpfs by default.  Potentially /sys/fs/cgroup will be showing up also.

Comment 19 Daniel Walsh 2015-04-23 17:53:20 UTC
I would like to have a meeting to discuss the IPA Image/Images to make sure they follow the latest ways we are thinking in the AppInfra team.

Comment 25 Fedora Update System 2015-04-24 17:51:30 UTC
docker-io-1.6.0-2.git3eac457.fc21 has been submitted as an update for Fedora 21.

Comment 27 Fedora Update System 2015-04-26 12:53:52 UTC
docker-io-1.6.0-2.git3eac457.fc21 has been pushed to the Fedora 21 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.