Bug 1498281

Do you have oci-umount installed?

oci-umount-2.0.0-2.gitf90b64c.fc26.x86_64

This is on Fedora Atomic Host with `atomic host upgrade` as of 2017-10-03 PM.


   # atomic host status                                                                                  
   State: idle                                                                                                                        
   Deployments:
   ? fedora-atomic:fedora/26/x86_64/atomic-host
                      Version: 26.131 (2017-09-19 22:29:04)
                       Commit: 98088cb6ed2a4b3f7e4e7bf6d34f9e137c296bc43640b4c1967631f22fe1802f
                 GPGSignature: Valid signature by E641850B77DF435378D1D7E2812A6B4B64DAB85D

     fedora-atomic:fedora/26/x86_64/atomic-host
                      Version: 26.83 (2017-07-05 13:04:12)
                       Commit: 2e8d8dab1c794ff61ba79e60652a687cd1b3c43ad97f0aac38fd271553e58ee2
                 GPGSignature: Valid signature by E641850B77DF435378D1D7E2812A6B4B64DAB85D

I greatly regret that I am going offline now and will not be able to reply again until, at the hopeful earliest, Friday.

cat /etc/oci-umount.conf

So there are two paths inside container. /var/lib/docker/overlay2 got unmounted but /rootfs/....../overlay2 did not. 

I think oci-umount logic expected /rootfs/var/lib/docker/overlay2 to be present because atomic does so much of magic with mounts, its not there. Instead, "/rootfs/sysroot/ostree/deploy/fedora-atomic/var/lib/docker/overlay2", is there. 

But this sounds like an atomic problem. If I do "-v /:/rootfs", shouldn't I see a mount point on /rootfs/var/lib/docker/overlay2 inside container?

Ed, please also paste journalctl logs which are prefixed with umounthook.

I think the /rootfs/sysroot/ostree/deploy/fedora-atomic/var/lib/docker/overlay2 is unrelated and is not a leaked mount point into this container. It is a mount point on the host, but unrelated to the container.

/etc/oci-umount.conf:

   # This contains a list of paths on host which will be unmounted inside                                                          
   # container. (If they are mounted inside container).                                                                            
                                                                                                                                   
   # If there is a "/*" at the end, that means only mounts underneath that                                                         
   # mounts (submounts) will be unmounted but top level mount will remain                                                          
   # in place.                                                                                                                     
   /var/lib/docker/overlay2                                                                                                        
   /var/lib/docker/overlay                                                                                                         
   /var/lib/docker/devicemapper                                                                                                    
   /var/lib/docker/containers/*                                                                                                    
   /var/lib/docker-latest/overlay2                                                                                                 
   /var/lib/docker-latest/overlay                                                                                                  
   /var/lib/docker-latest/devicemapper                                                                                             
   /var/lib/docker-latest/containers/*                                                                                             
   /var/lib/containers/storage/lvm                                                                                                 
   /var/lib/containers/storage/devicemapper                                                                                        
   /var/lib/containers/storage/overlay                                                                                             
   /var/run/containers/storage

Created attachment 1335366 [details]
journalctl -f | grep umounthook

Attached: journalctl -f | grep umounthook

command run in other window: docker run --rm -v /:/rootfs -v /var/lib/docker:/var/lib/docker docker.io/fedora:latest findmnt -R -n --list / |grep /overlay2

Ed is this still the case with the latest oci-umount?

Dan: yes, it looks like it:

    # docker run --rm -v /:/rootfs -v /var/lib/docker:/var/lib/docker docker.io/fedora:latest findmnt -R -n --list / |grep /overlay2
    /                                                                     overlay                                                                                                                                                        overlay    rw,relatime,context="system_u:object_r:container_file_t:s0:c9,c93",lowerdir=/var/lib/docker/overlay2/l/KNWJRV376GNMPNXU2SMUUJGH2F:/var/lib/docker/overlay2/l/THD7KDPGXPUUNNDWWYLPZXM2TJ,upperdir=/var/lib/docker/overlay2/11f60c6ee54e427f4cfd20e30c6a24fb38cdc0e1e745fcf5bb2196344b73fb2d/diff,workdir=/var/lib/docker/overlay2/11f60c6ee54e427f4cfd20e30c6a24fb38cdc0e1e745fcf5bb2196344b73fb2d/work
    /rootfs/sysroot/ostree/deploy/fedora-atomic/var/lib/docker/overlay2   /dev/mapper/atomicos-root[/ostree/deploy/fedora-atomic/var/lib/docker/overlay2]                                                                                xfs        rw,relatime,seclabel,attr2,inode64,noquota

    # docker run --rm -v /:/rootfs -v /var/lib/docker:/var/lib/docker docker.io/fedora:latest ls -l /rootfs/sysroot/ostree/deploy/fedora-atomic/var/lib/docker/overlay2
    ls: cannot access '/rootfs/sysroot/ostree/deploy/fedora-atomic/var/lib/docker/overlay2/backingFsBlockDev': Permission denied
    total 0
    drwx------. 5 root root  69 Jan 22 21:37 5ddb64baba16bf80ed436f40b46c3e7e0afaf195fb41c397290249698f72e311
    drwx------. 5 root root  69 Jan 22 21:37 5ddb64baba16bf80ed436f40b46c3e7e0afaf195fb41c397290249698f72e311-init
    b?????????? ? ?    ?      ?            ? backingFsBlockDev
    drwx------. 3 root root  30 Jan 22 21:35 d254fd08ae8efabbfcd5d5c5f4e98457b15dd921466758f2a13cd0eb4502c8b3
    drwx------. 2 root root 108 Jan 22 21:37 l

host details:

    # atomic host status
    State: idle
    Deployments:
    ? fedora-atomic:fedora/27/x86_64/atomic-host
                       Version: 27.61 (2018-01-17 15:52:47)
                        Commit: 772ab185b0752b0d6bc8b2096d08955660d80ed95579e13e136e6a54e3559ca9
                  GPGSignature: Valid signature by 860E19B0AFA800A1751881A6F55E7430F5282EE4

rpms:

    docker-1.13.1-44.git584d391.fc27.x86_64
    oci-umount-2.3.2-1.git3025b19.fc27.x86_64

oci-umount.conf :

    # This contains a list of paths on host which will be unmounted inside
    # container. (If they are mounted inside container).
    
    # If there is a "/*" at the end, that means only mounts underneath that
    # mounts (submounts) will be unmounted but top level mount will remain
    # in place.
    /var/lib/docker/overlay2
    /var/lib/docker/overlay
    /var/lib/docker/devicemapper
    /var/lib/docker/containers/*
    /var/lib/docker-latest/overlay2
    /var/lib/docker-latest/overlay
    /var/lib/docker-latest/devicemapper
    /var/lib/docker-latest/containers/*

Vivek thoughts?

I think it is a problem where source mount is visible on host in multiple paths and we don't have any mechanism to deal with it. For example, on a regular fedora host, do following.

# mkdir /root/foo/foo2
# mount --rbind /var/lib/  /root/foo/foo2
# docker run -ti -v /:/rootfs fedora bash

Now inside the container you will still see the overlay2 mount through /rootfs/foo/foo2/docker/ovlerlay2

oci-umount has already taken out two mounts. One at /var/lib/docker/overlay2
and another one at /rootfs/var/lib/docker/overlay2. But it does not know anything about source path on host "/root/foo/foo2/docker/overlay2" and when that path leaks inside container due to "-v /:/rootfs", oci-umount can't do anything about it.

To get rid of it, oci-umount will have to be told about it explicitly. So put something like this in /etc/oci-umount.conf

/root/foo/foo2/docker/overlay2

Similarly for atomic case, if we put something like this in oci-umount.conf, that should take care of it.

/sysroot/ostree/deploy/fedora-atomic/var/lib/docker/overlay2


Ed, can you try modifying /etc/oci-umount.conf and put above and see if it helps.

Yes, it helps:

    # docker run --rm -v /:/rootfs -v /var/lib/docker:/var/lib/docker docker.io/fedora:latest findmnt -R -n --list / |grep /overlay2
    /                                                                     overlay                                                                                                                                                        overlay    rw,relatime,context="system_u:object_r:container_file_t:s0:c297,c565",lowerdir=/var/lib/docker/overlay2/l/MU22NS3P366FKOLXVPQSWEDPEH:/var/lib/docker/overlay2/l/EQG7KJX7LIONH4KNW4W6QO4CYC,upperdir=/var/lib/docker/overlay2/1f209110ec4de4a7a504f278b2c6c5fb47650909b36ad9be87f488d89ec00197/diff,workdir=/var/lib/docker/overlay2/1f209110ec4de4a7a504f278b2c6c5fb47650909b36ad9be87f488d89ec00197/work

But as you imply, this is just a workaround. The real issue, as I see it, is that oci-umount works on a blacklist basis ("everything is OK except these few") -- there will always be ways to bypass that, almost always unintentionally.

I think that's the right fix. (its not a workaround).

No, oci-umount does not work on blacklist basis. It just works on the list of paths (source paths on host) in /etc/oci-umount.conf and if that path is mounted inside container, it unmounts that. It does not care about other mounts other than listed in /etc/oci-umount.conf.

/sysroot/ostree/deploy/fedora-atomic/var/lib/docker/overlay2 is not same as /var/lib/docker/overlay2. So we will have to explicitly specify it in /etc/oci-umount.conf.

I think atomic kickstart might have to be modified for this.

Colin, WDYT.

The /sysroot thing seems like a plain ostree bug - offhand, I think we should be making /sysroot slave and not shared.  Let me do some testing.

Upstream PR: https://github.com/ostreedev/ostree/pull/1438

Since this is fixed upstream, we don't need to fix this in oci-umount? Is that correct?  Do we have this issue in RHEL 7?