Bug 1381696

Not sure if there is a plan to fix this code, but I just create this BZ for tracking and document as a known issue in 7.3 note.

Brent is this fixed in atomic-1.13?

No, I wasnt aware we support overlay2 now, do we?

Yes devicemapper,overlay and overlay2.  I fixed all of the atomic mount code,  I have a feeling that this will just work.

Any progress?

Should be fixed in atomic-1.14.

Unfortunately, it is still broken as version atomic-1.14.1-5.el7.x86_64

# atomic --debug scan --scanner openscap --scan_type cve  registry.access.redhat.com/rhel7
Created /run/atomic/2017-01-26-10-46-01-673026
docker run -t --rm -v /etc/localtime:/etc/localtime -v /run/atomic/2017-01-26-10-46-01-673026:/scanin -v /var/lib/atomic/openscap/2017-01-26-10-46-01-673026:/scanout:rw,Z -v /etc/oscapd:/etc/oscapd:ro rhel7/openscap oscapd-evaluate scan --no-standard-compliance --targets chroots-in-dir:///scanin --output /scanout -j1
Created /run/atomic/2017-01-26-10-46-01-673026/e4b79d4d89ab9b0aa873fd8dc99c652483bb645f317712ef30d5904ac2eafef3
Mounted {u'Created': 1484254315, u'Labels': {u'com.redhat.component': u'rhel-server-docker', u'authoritative-source-url': u'registry.access.redhat.com', u'distribution-scope': u'public', u'vendor': u'Red Hat, Inc.', u'Name': u'rhel7', u'io.k8s.display-name': u'Red Hat Enterprise Linux 7', u'description': u'The Red Hat Enterprise Linux Base image is designed to be a fully supported foundation for your containerized applications.  This base image provides your operations and application teams with the packages, language runtimes and tools necessary to run, maintain, and troubleshoot all of your applications. This image is maintained by Red Hat and updated regularly. It is designed and engineered to be the base layer for all of your containerized applications, middleware and utilites. When used as the source for all of your containers, only one copy will ever be downloaded and cached in your production environment. Use this image just like you would a regular Red Hat Enterprise Linux distribution. Tools like yum, gzip, and bash are provided by default. For further information on how this image was built look at the /root/anacanda-ks.cfg file.', u'summary': u'Provides the latest release of Red Hat Enterprise Linux 7 in a fully featured and supported base image.', u'vcs-type': u'git', u'name': u'rhel7', u'vcs-ref': u'06e55ffd458c665f861599ac9c7550a037d85ac7', u'release': u'66', u'Version': u'7.3', u'architecture': u'x86_64', u'version': u'7.3', u'Release': u'66', u'BZComponent': u'rhel-server-docker', u'build-date': u'2017-01-12T15:36:30.088642', u'io.openshift.tags': u'base rhel7', u'com.redhat.build-host': u'rcm-img-docker02.build.eng.bos.redhat.com'}, 'ImageId': u'e4b79d4d89ab9b0aa873fd8dc99c652483bb645f317712ef30d5904ac2eafef3', u'VirtualSize': 192540107, u'ParentId': u'', 'input': 'registry.access.redhat.com/rhel7', u'RepoTags': [u'registry.access.redhat.com/rhel7:latest'], u'RepoDigests': [u'registry.access.redhat.com/rhel7@sha256:0614d58c96e8d1a04a252880a6c33b48b4685cafae048a70dd9e821edf62cab9'], u'Id': u'e4b79d4d89ab9b0aa873fd8dc99c652483bb645f317712ef30d5904ac2eafef3', 'ImageType': 'Docker', u'Size': 192540107} to /run/atomic/2017-01-26-10-46-01-673026/e4b79d4d89ab9b0aa873fd8dc99c652483bb645f317712ef30d5904ac2eafef3
Creating the output dir at /var/lib/atomic/openscap/2017-01-26-10-46-01-673026
Unable to find image 'rhel7/openscap:latest' locally
Trying to pull repository registry.access.redhat.com/rhel7/openscap ... 
sha256:db1206c551c8117a53cb89c4f3ccd6072d2e03e5d6d0ad029b751864429dbd55: Pulling from registry.access.redhat.com/rhel7/openscap
7bd78273b666: Already exists 
c196631bd9ac: Already exists 
db7cef4d643b: Pull complete 
e3b3e87ac388: Pull complete 
Digest: sha256:db1206c551c8117a53cb89c4f3ccd6072d2e03e5d6d0ad029b751864429dbd55
Status: Downloaded newer image for registry.access.redhat.com/rhel7/openscap:latest
INFO:OpenSCAP Daemon one-off evaluator 0.1.6
INFO:Autodetected "oscap" in path "/usr/bin/oscap".
INFO:Autodetected "oscap-ssh" in path "/usr/bin/oscap-ssh".
INFO:Autodetected "oscap-vm" in path "/usr/bin/oscap-vm".
INFO:Autodetected "oscap-docker" in path "/usr/bin/oscap-docker".
INFO:Autodetected "oscap-chroot" in path "/usr/bin/oscap-chroot".
WARNING:Can't import the 'docker' package. Container scanning functionality will be disabled.
INFO:Autodetected SCAP content at "/usr/share/openscap/cpe/openscap-cpe-oval.xml".
INFO:Autodetected SCAP content in path "/usr/share/xml/scap/ssg/content".
INFO:Creating tasks directory at '/var/lib/oscapd/tasks' because it didn't exist.
INFO:Creating results directory at '/var/lib/oscapd/results' because it didn't exist.
INFO:Creating results work in progress directory at '/var/lib/oscapd/work_in_progress' because it didn't exist.
INFO:Evaluated EvaluationSpec, exit_code=0.
INFO:Had a local version of /var/lib/oscapd/cve_feeds/com.redhat.rhsa-RHEL7.xml but it wasn't new enough
INFO:Evaluated EvaluationSpec, exit_code=0.
INFO:[100.00%] Scanned target 'chroot:///scanin/e4b79d4d89ab9b0aa873fd8dc99c652483bb645f317712ef30d5904ac2eafef3'

registry.access.redhat.com/rhel7 (e4b79d4d89ab9b0)

registry.access.redhat.com/rhel7 passed the scan

Files associated with this scan are in /var/lib/atomic/openscap/2017-01-26-10-46-01-673026.

The device mounted at /run/atomic/2017-01-26-10-46-01-673026/e4b79d4d89ab9b0aa873fd8dc99c652483bb645f317712ef30d5904ac2eafef3 is not a docker container.
Traceback (most recent call last):
  File "/bin/atomic", line 187, in <module>
    sys.exit(_func())
  File "/usr/lib/python2.7/site-packages/Atomic/scan.py", line 174, in scan
    self._unmount_rootfs_in_dir()
  File "/usr/lib/python2.7/site-packages/Atomic/scan.py", line 253, in _unmount_rootfs_in_dir
    self.unmount(rootfs_dir)
  File "/usr/lib/python2.7/site-packages/Atomic/scan.py", line 461, in unmount
    m.unmount()
  File "/usr/lib/python2.7/site-packages/Atomic/mount.py", line 210, in unmount
    raise ValueError(dme)
ValueError: The device mounted at /run/atomic/2017-01-26-10-46-01-673026/e4b79d4d89ab9b0aa873fd8dc99c652483bb645f317712ef30d5904ac2eafef3 is not a docker container.

This is probably a different issue.

Could you see if atomic mount works with overlay2?

# atomic mount --storage docker registry.access.redhat.com/rhel7/openscap /mnt/

# atomic umount /mnt/
The device mounted at /mnt/ is not a docker container.

# ls /mnt/
bin   dev  home  lib64       media  opt   root  sbin  sys  usr
boot  etc  lib   lost+found  mnt    proc  run   srv   tmp  var

# mount | grep overlay
/dev/vda1 on /var/lib/docker-latest/overlay2 type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
overlay on /run/atomic/2017-01-26-10-46-01-673026/e4b79d4d89ab9b0aa873fd8dc99c652483bb645f317712ef30d5904ac2eafef3 type overlay (ro,relatime,seclabel,lowerdir=/var/lib/docker-latest/overlay2/463b4fd8f57c9500b213eb58264288247c442a01647985f4af77b9641c044397-init/diff:/var/lib/docker-latest/overlay2/ce73c6685a214a803fb20508ddfabd283ba016d898506bb564eb64b12f0bf498/diff:/var/lib/docker-latest/overlay2/4c2009b40a94e8f81e1afd7e3f52d91f4b55e294dc500c547635cc582e298dd1/diff,upperdir=/var/lib/docker-latest/overlay2/463b4fd8f57c9500b213eb58264288247c442a01647985f4af77b9641c044397/diff,workdir=/var/lib/docker-latest/overlay2/463b4fd8f57c9500b213eb58264288247c442a01647985f4af77b9641c044397/work)
overlay on /run/atomic/2017-01-26-10-48-12-762259/e4b79d4d89ab9b0aa873fd8dc99c652483bb645f317712ef30d5904ac2eafef3 type overlay (ro,relatime,seclabel,lowerdir=/var/lib/docker-latest/overlay2/c4f640ff1809e085931c8ae20ccadb13f8f013cfe05290c5527b258e929fa92e-init/diff:/var/lib/docker-latest/overlay2/ce73c6685a214a803fb20508ddfabd283ba016d898506bb564eb64b12f0bf498/diff:/var/lib/docker-latest/overlay2/4c2009b40a94e8f81e1afd7e3f52d91f4b55e294dc500c547635cc582e298dd1/diff,upperdir=/var/lib/docker-latest/overlay2/c4f640ff1809e085931c8ae20ccadb13f8f013cfe05290c5527b258e929fa92e/diff,workdir=/var/lib/docker-latest/overlay2/c4f640ff1809e085931c8ae20ccadb13f8f013cfe05290c5527b258e929fa92e/work)
overlay on /mnt type overlay (ro,relatime,seclabel,lowerdir=/var/lib/docker-latest/overlay2/c450efdb4a3396d02a26347230b2855308d85cc130923faee1a3c6c75936d332-init/diff:/var/lib/docker-latest/overlay2/85fdd0a227fa59ade8f738ce97ba473a4e110de802a331bb0911b1c5e8ded468/diff:/var/lib/docker-latest/overlay2/4905133fc0e56b008a6efceebd02e8ad236a8142540dcdf93825c98177eadcc3/diff:/var/lib/docker-latest/overlay2/ce73c6685a214a803fb20508ddfabd283ba016d898506bb564eb64b12f0bf498/diff:/var/lib/docker-latest/overlay2/4c2009b40a94e8f81e1afd7e3f52d91f4b55e294dc500c547635cc582e298dd1/diff,upperdir=/var/lib/docker-latest/overlay2/c450efdb4a3396d02a26347230b2855308d85cc130923faee1a3c6c75936d332/diff,workdir=/var/lib/docker-latest/overlay2/c450efdb4a3396d02a26347230b2855308d85cc130923faee1a3c6c75936d332/work)

It looks atomic umount NOT work well with overlay2.

On the other hand, umount works fine.

# umount /mnt
# umount /run/atomic/2017-01-26-10-48-12-762259/e4b79d4d89ab9b0aa873fd8dc99c652483bb645f317712ef30d5904ac2eafef3
# umount /run/atomic/2017-01-26-10-48-12-762259/e4b79d4d89ab9b0aa873fd8dc99c652483bb645f317712ef30d5904ac2eafef3
# mount | grep overlay
/dev/vda1 on /var/lib/docker-latest/overlay2 type xfs (rw,relatime,seclabel,attr2,inode64,noquota)

OK, I found culprit. There is several hard-coded /var/lib/docker/ places in Atomic/mount.py which won't work with docker-latest. Once I fixed them, it works fine again. We will just need a patch to copy with both docker and docker-latest.

# diff -u /usr/lib/python2.7/site-packages/Atomic/mount.py.orig /usr/lib/python2.7/site-packages/Atomic/mount.py
--- /usr/lib/python2.7/site-packages/Atomic/mount.py.orig	2017-01-26 12:28:26.717999574 -0500
+++ /usr/lib/python2.7/site-packages/Atomic/mount.py	2017-01-26 12:29:40.449999574 -0500
@@ -403,15 +403,15 @@
 
     @staticmethod
     def _no_gd_api_dm(cid):
-        desc_file = os.path.join('/var/lib/docker/devicemapper/metadata', cid)
+        desc_file = os.path.join('/var/lib/docker-latest/devicemapper/metadata', cid)
         desc = json.loads(open(desc_file).read())
         return desc['device_id'], desc['size']
 
     @staticmethod
     def _no_gd_api_overlay(cid, driver):
-        prefix = os.path.join('/var/lib/docker/%s/' % driver, cid)
+        prefix = os.path.join('/var/lib/docker-latest/%s/' % driver, cid)
         ld_metafile = open(os.path.join(prefix, 'lower-id'))
-        ld_loc = os.path.join('/var/lib/docker/%s/' % driver, ld_metafile.read())
+        ld_loc = os.path.join('/var/lib/docker-latest/%s/' % driver, ld_metafile.read())
         return (os.path.join(ld_loc, 'root'), os.path.join(prefix, 'upper'),
                 os.path.join(prefix, 'work'))
 
@@ -679,7 +679,7 @@
         upperdir = [o.replace('upperdir=', '') for o in optstring.split(',')
                     if o.startswith('upperdir=')][0]
         cdir = upperdir.rsplit('/', 1)[0]
-        if not cdir.startswith('/var/lib/docker/%s/' % driver ):
+        if not cdir.startswith('/var/lib/docker-latest/%s/' % driver ):
             raise MountError('The device mounted at %s is not a '
                              'docker container.' % self.mountpoint )

I have created a pull request which should fix this.  I tested locally with overlay2.  Could you please test it and provide feedback?

The code can be found at:

https://github.com/projectatomic/atomic/pull/852

Let me know if you are able to test this.

Unfortunately, it is still broken with overlay + docker-latest after applied the patch at /usr/lib/python2.7/site-packages/Atomic/mount.py

# atomic --debug scan --scanner openscap --scan_type cve registry.access.redhat.com/rhel7
Created /run/atomic/2017-01-30-15-40-08-703680
docker run -t --rm -v /etc/localtime:/etc/localtime -v /run/atomic/2017-01-30-15-40-08-703680:/scanin -v /var/lib/atomic/openscap/2017-01-30-15-40-08-703680:/scanout:rw,Z -v /etc/oscapd:/etc/oscapd:ro rhel7/openscap oscapd-evaluate scan --no-standard-compliance --targets chroots-in-dir:///scanin --output /scanout -j1
Created /run/atomic/2017-01-30-15-40-08-703680/e4b79d4d89ab9b0aa873fd8dc99c652483bb645f317712ef30d5904ac2eafef3
Mounted {u'Created': 1484254315, u'Labels': {u'com.redhat.component': u'rhel-server-docker', u'authoritative-source-url': u'registry.access.redhat.com', u'distribution-scope': u'public', u'vendor': u'Red Hat, Inc.', u'Name': u'rhel7', u'io.k8s.display-name': u'Red Hat Enterprise Linux 7', u'description': u'The Red Hat Enterprise Linux Base image is designed to be a fully supported foundation for your containerized applications.  This base image provides your operations and application teams with the packages, language runtimes and tools necessary to run, maintain, and troubleshoot all of your applications. This image is maintained by Red Hat and updated regularly. It is designed and engineered to be the base layer for all of your containerized applications, middleware and utilites. When used as the source for all of your containers, only one copy will ever be downloaded and cached in your production environment. Use this image just like you would a regular Red Hat Enterprise Linux distribution. Tools like yum, gzip, and bash are provided by default. For further information on how this image was built look at the /root/anacanda-ks.cfg file.', u'summary': u'Provides the latest release of Red Hat Enterprise Linux 7 in a fully featured and supported base image.', u'vcs-type': u'git', u'name': u'rhel7', u'vcs-ref': u'06e55ffd458c665f861599ac9c7550a037d85ac7', u'release': u'66', u'Version': u'7.3', u'architecture': u'x86_64', u'version': u'7.3', u'Release': u'66', u'BZComponent': u'rhel-server-docker', u'build-date': u'2017-01-12T15:36:30.088642', u'io.openshift.tags': u'base rhel7', u'com.redhat.build-host': u'rcm-img-docker02.build.eng.bos.redhat.com'}, 'ImageId': u'e4b79d4d89ab9b0aa873fd8dc99c652483bb645f317712ef30d5904ac2eafef3', u'VirtualSize': 192532823, u'ParentId': u'', 'input': 'registry.access.redhat.com/rhel7', u'RepoTags': [u'registry.access.redhat.com/rhel7:latest'], u'RepoDigests': [u'registry.access.redhat.com/rhel7@sha256:0614d58c96e8d1a04a252880a6c33b48b4685cafae048a70dd9e821edf62cab9'], u'Id': u'e4b79d4d89ab9b0aa873fd8dc99c652483bb645f317712ef30d5904ac2eafef3', 'ImageType': 'Docker', u'Size': 192532823} to /run/atomic/2017-01-30-15-40-08-703680/e4b79d4d89ab9b0aa873fd8dc99c652483bb645f317712ef30d5904ac2eafef3
Creating the output dir at /var/lib/atomic/openscap/2017-01-30-15-40-08-703680
INFO:OpenSCAP Daemon one-off evaluator 0.1.6
INFO:Autodetected "oscap" in path "/usr/bin/oscap".
INFO:Autodetected "oscap-ssh" in path "/usr/bin/oscap-ssh".
INFO:Autodetected "oscap-vm" in path "/usr/bin/oscap-vm".
INFO:Autodetected "oscap-docker" in path "/usr/bin/oscap-docker".
INFO:Autodetected "oscap-chroot" in path "/usr/bin/oscap-chroot".
WARNING:Can't import the 'docker' package. Container scanning functionality will be disabled.
INFO:Autodetected SCAP content at "/usr/share/openscap/cpe/openscap-cpe-oval.xml".
INFO:Autodetected SCAP content in path "/usr/share/xml/scap/ssg/content".
INFO:Creating tasks directory at '/var/lib/oscapd/tasks' because it didn't exist.
INFO:Creating results directory at '/var/lib/oscapd/results' because it didn't exist.
INFO:Creating results work in progress directory at '/var/lib/oscapd/work_in_progress' because it didn't exist.
INFO:Evaluated EvaluationSpec, exit_code=0.
INFO:Had a local version of /var/lib/oscapd/cve_feeds/com.redhat.rhsa-RHEL7.xml but it wasn't new enough
INFO:Evaluated EvaluationSpec, exit_code=0.
INFO:[100.00%] Scanned target 'chroot:///scanin/e4b79d4d89ab9b0aa873fd8dc99c652483bb645f317712ef30d5904ac2eafef3'

registry.access.redhat.com/rhel7 (e4b79d4d89ab9b0)

registry.access.redhat.com/rhel7 passed the scan

Files associated with this scan are in /var/lib/atomic/openscap/2017-01-30-15-40-08-703680.

The device mounted at /run/atomic/2017-01-30-15-40-08-703680/e4b79d4d89ab9b0aa873fd8dc99c652483bb645f317712ef30d5904ac2eafef3 is not a docker container.
Traceback (most recent call last):
  File "/bin/atomic", line 187, in <module>
    sys.exit(_func())
  File "/usr/lib/python2.7/site-packages/Atomic/scan.py", line 174, in scan
    self._unmount_rootfs_in_dir()
  File "/usr/lib/python2.7/site-packages/Atomic/scan.py", line 253, in _unmount_rootfs_in_dir
    self.unmount(rootfs_dir)
  File "/usr/lib/python2.7/site-packages/Atomic/scan.py", line 461, in unmount
    m.unmount()
  File "/usr/lib/python2.7/site-packages/Atomic/mount.py", line 210, in unmount
    raise ValueError(dme)
ValueError: The device mounted at /run/atomic/2017-01-30-15-40-08-703680/e4b79d4d89ab9b0aa873fd8dc99c652483bb645f317712ef30d5904ac2eafef3 is not a docker container.

can I get access to this vm/machine?  im not able to replicate anymore with the latest code.

Brent, does atomic-1.15.2-4.el7.x86_64 in atomic host 7.3.3 include the fix for this?

It should.  I have been testing against upstream so I cannot speak to those specific versions.  Are you able to test it for me and determine if it works?

lets assume it does and let QE prove us wrong...

Fixed in atomic-1.15.2-4.el7.x86_64

Unfortunately, it is still broken with overlay2 + docker-latest.

The device mounted at /run/atomic/2017-02-23-16-41-15-099039/e4b79d4d89ab9b0aa873fd8dc99c652483bb645f317712ef30d5904ac2eafef3 is not a docker container.
Traceback (most recent call last):
  File "/bin/atomic", line 188, in <module>
    sys.exit(_func())
  File "/usr/lib/python2.7/site-packages/Atomic/scan.py", line 174, in scan
    self._unmount_rootfs_in_dir()
  File "/usr/lib/python2.7/site-packages/Atomic/scan.py", line 253, in _unmount_rootfs_in_dir
    self.unmount(rootfs_dir)
  File "/usr/lib/python2.7/site-packages/Atomic/scan.py", line 461, in unmount
    m.unmount()
  File "/usr/lib/python2.7/site-packages/Atomic/mount.py", line 210, in unmount
    raise ValueError(dme)
ValueError: The device mounted at /run/atomic/2017-02-23-16-41-15-099039/e4b79d4d89ab9b0aa873fd8dc99c652483bb645f317712ef30d5904ac2eafef3 is not a docker container.

can you provide access details so I can verify this and/or a pointer to the image you used? I need to be able to replicate this exactly.

Sent you an email with the access detail.

This works with 1.15.3

[cloud-user@localhost ~]$ sudo atomic mount registry.access.redhat.com/rhel7 /tmp/foo
[cloud-user@localhost ~]$ sudo atomic unmount /tmp/foo
[cloud-user@localhost ~]$ rpm -q docker-latest
docker-latest-1.12.6-10.el7.x86_64
[cloud-user@localhost ~]$ atomic -v
1.15.3

Well, it can only be reproduced if you used overlay or overlay2.

I believe that https://github.com/projectatomic/atomic/pull/912 fixes this.

Fixed in atomic-1.16.2

Since there is no rpm build with it in brew yet, it probably better to move to POST instead of MODIFIED.

I always move things to modified.  Never used POST.  Does this mean fixed in the next release?

In RHEL CDW, it usually use,

POST: the patch has been posted upstream.
MODIFIED: the rpm has been build in brew.
ON_QA: the build has been attached to the erratum.

Not a big deal if you want to use something different in Agile mode.

Lokesh, would you like to add this to the 7.3.4 atomic errata?