1368566 – atomic scan does not work with overlayfs

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1368566 - atomic scan does not work with overlayfs

Summary: atomic scan does not work with overlayfs

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	atomic
Sub Component:
Version:	7.2
Hardware:	x86_64
OS:	Linux
Priority:	high
Severity:	high
Target Milestone:	rc
Target Release:	---
Assignee:	Brent Baude
QA Contact:	atomic-bugs@redhat.com
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2016-08-19 18:53 UTC by Qian Cai
Modified:	2016-11-04 09:06 UTC (History)
CC List:	5 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2016-11-04 09:06:41 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2016:2628	0	normal	SHIPPED_LIVE	atomic bug fix and enhancement update	2016-11-03 18:17:14 UTC

Description Qian Cai 2016-08-19 18:53:55 UTC

Description of problem:
# atomic install rhel7/openscap
Using default tag: latest
Trying to pull repository registry.access.redhat.com/rhel7/openscap ... 
Pulling repository registry.access.redhat.com/rhel7/openscap
d71330f48eb3: Pull complete 
6f7a31562d1e: Pull complete 
Status: Downloaded newer image for registry.access.redhat.com/rhel7/openscap:latest
registry.access.redhat.com/rhel7/openscap: this image was pulled from a legacy registry.  Important: This registry version will not be supported in future versions of docker.
docker run --rm --privileged -v /:/host/ rhel7/openscap sh /root/install.sh

Installing the configuration file 'openscap' into /etc/atomic.d/.  You can now use this scanner with atomic scan with the --scanner openscap command-line option.  You can also set 'openscap' as the default scanner in /etc/atomic.conf.  To list the scanners you have configured for your system, use 'atomic scan --list'.

Saving current config.ini as config.ini.2016-08-19-14:43:06.atomic_save
Updating config.ini with latest configuration
Installation complete. You can customize /etc/oscapd/config.ini as needed.

# atomic scan rhel7
Unable to associate 'rhel7' with an image or container

# docker images
REPOSITORY                                  TAG                 IMAGE ID            CREATED             SIZE
registry.access.redhat.com/rhel7/openscap   latest              e86fecac7d50        3 weeks ago         362.3 MB

# docker pull rhel7
Using default tag: latest
Trying to pull repository registry.access.redhat.com/rhel7 ... 
Pulling repository registry.access.redhat.com/rhel7
6f7a31562d1e: Already exists 
Status: Image is up to date for registry.access.redhat.com/rhel7:latest
registry.access.redhat.com/rhel7: this image was pulled from a legacy registry.  Important: This registry version will not be supported in future versions of docker.

# docker images
REPOSITORY                                  TAG                 IMAGE ID            CREATED             SIZE
registry.access.redhat.com/rhel7/openscap   latest              e86fecac7d50        3 weeks ago         362.3 MB
registry.access.redhat.com/rhel7            latest              4a6b6e1a17d7        3 weeks ago         201.6 MB

# atomic scan rhel7
[Errno 2] No such file or directory: u'/var/lib/docker/overlay/d4e51719a572f40e7408d5b3f7192d275505544809b46f670977d6b1170ea44f/lower-id'

Version-Release number of selected component (if applicable):
atomic-1.10.5-7.el7.x86_64
docker-1.10.3-46.el7.10.x86_64
rhel atomic host 7.2.6

How reproducible:
always

Comment 1 Qian Cai 2016-08-19 19:04:20 UTC

This seems also caused atomic scan failed when in atomic devmode.

# atomic host unlock
Development mode enabled. A writable overlayfs is now mounted on /usr.
All changes there will be discarded on reboot.

# atomic install rhel7/openscap
Using default tag: latest
Trying to pull repository registry.access.redhat.com/rhel7/openscap ...
Pulling repository registry.access.redhat.com/rhel7/openscap
d71330f48eb3: Pull complete
6f7a31562d1e: Pull complete
Status: Downloaded newer image for registry.access.redhat.com/rhel7/openscap:latest
registry.access.redhat.com/rhel7/openscap: this image was pulled from a legacy registry. Important: This registry version will not be supported in future versions of docker.
docker run --rm --privileged -v /:/host/ rhel7/openscap sh /root/install.sh

Installing the configuration file 'openscap' into /etc/atomic.d/. You can now use this scanner with atomic scan with the --scanner openscap command-line option. You can also set 'openscap' as the default scanner in /etc/atomic.conf. To list the scanners you have configured for your system, use 'atomic scan --list'.

Saving current config.ini as config.ini.2016-08-19-14:57:37.atomic_save
Updating config.ini with latest configuration
Installation complete. You can customize /etc/oscapd/config.ini as needed.

# atomic scan rhel7
Unable to associate 'rhel7' with an image or container
bash-4.2# docker pull rhel7
Using default tag: latest
Trying to pull repository registry.access.redhat.com/rhel7 ...
Pulling repository registry.access.redhat.com/rhel7
6f7a31562d1e: Already exists
Status: Image is up to date for registry.access.redhat.com/rhel7:latest
registry.access.redhat.com/rhel7: this image was pulled from a legacy registry. Important: This registry version will not be supported in future versions of docker.

# atomic scan rhel7
docker run -it --rm -v /etc/localtime:/etc/localtime -v /run/atomic/2016-08-19-19-02-54-430358:/scanin -v /var/lib/atomic/openscap/2016-08-19-19-02-54-430358:/scanout:rw,Z -v /etc/oscapd:/etc/oscapd:ro rhel7/openscap oscapd-evaluate scan --no-standard-compliance --targets chroots-in-dir:///scanin --output /scanout

rhel7 (sha256:4a6b6e1a)

rhel7 is not supported for this scan.

Files associated with this scan are in /var/lib/atomic/openscap/2016-08-19-19-02-54-430358.

Comment 3 Qian Cai 2016-08-19 19:44:06 UTC

Same issue even with xfs ftype=1.

# xfs_info /var/lib/docker
meta-data=/dev/vdb               isize=512    agcount=4, agsize=655360 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0 spinodes=0
data     =                       bsize=4096   blocks=2621440, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal               bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Comment 4 Daniel Walsh 2016-08-20 09:24:35 UTC

I made a lot of fixes in upstream master to handle overlay.  Perhaps we need to cut a new version of atomic, to grab these.  atomic-1.11.1?

Comment 5 Daniel Walsh 2016-08-26 18:52:58 UTC

If you are running with SELinux this will also not work. Since unlock and SELinux do not play well together.

Comment 6 Daniel Walsh 2016-08-26 18:53:36 UTC

Fixed in atomic-1.11 with SELinux disabled or on a RHEL system

Comment 8 Alex Jia 2016-09-18 06:12:58 UTC

atomic scan works w/ overlayfs when the SELinux is Permissive mode.

[cloud-user@atomic-host-001 atomic]$ atomic host status
State: idle
Deployments:
● rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard
       Version: 7.2.7 (2016-09-09 18:43:35)
        Commit: 347c3f5eb641e69fc602878c646cf42c4bcd5d9f36847a1f24ff8f3ec80f17b1
        OSName: rhel-atomic-host
      Unlocked: development

  rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard
       Version: 7.2.7 (2016-09-08 17:14:40)
        Commit: a018354891f8d991c5cf12962907d54231c7273508f046161e1699b734738d1f
        OSName: rhel-atomic-host

[cloud-user@atomic-host-001 atomic]$ sudo setenforce 0
[cloud-user@atomic-host-001 atomic]$ getenforce
Permissive

[cloud-user@atomic-host-001 atomic]$ sudo atomic scan --scanner openscap --scan_type cve registry.access.redhat.com/rhel7:latest
docker run -it --rm -v /etc/localtime:/etc/localtime -v /run/atomic/2016-09-18-03-53-16-673435:/scanin -v /var/lib/atomic/openscap/2016-09-18-03-53-16-673435:/scanout:rw,Z -v /etc/oscapd:/etc/oscapd:ro rhel7/openscap oscapd-evaluate scan --no-standard-compliance --targets chroots-in-dir:///scanin --output /scanout

registry.access.redhat.com/rhel7:latest (sha256:98a88a8b)

registry.access.redhat.com/rhel7:latest passed the scan

Files associated with this scan are in /var/lib/atomic/openscap/2016-09-18-03-53-16-673435.


Just a record, w/ SELinux enforcing mode, atomic scan doesn't support scanning image.

[cloud-user@atomic-host-001 atomic]$ getenforce
Enforcing

[cloud-user@atomic-host-001 atomic]$ sudo atomic scan --scanner openscap --scan_type cve registry.access.redhat.com/rhel7
docker run -it --rm -v /etc/localtime:/etc/localtime -v /run/atomic/2016-09-18-03-52-27-776245:/scanin -v /var/lib/atomic/openscap/2016-09-18-03-52-27-776245:/scanout:rw,Z -v /etc/oscapd:/etc/oscapd:ro rhel7/openscap oscapd-evaluate scan --no-standard-compliance --targets chroots-in-dir:///scanin --output /scanout

registry.access.redhat.com/rhel7 (sha256:98a88a8b)

     registry.access.redhat.com/rhel7 is not supported for this scan.

Files associated with this scan are in /var/lib/atomic/openscap/2016-09-18-03-52-27-776245.

Comment 9 Qian Cai 2016-10-04 18:27:03 UTC

(In reply to Alex Jia from comment #8)
> Just a record, w/ SELinux enforcing mode, atomic scan doesn't support
> scanning image.
This looks like a bug to me. atomic scan should work with overlayfs and host's selinux enabled.

Comment 10 Daniel Walsh 2016-10-04 18:51:54 UTC

We don't support OverlayFS and SELinux at this point so why is this a bug?

Comment 11 Qian Cai 2016-10-04 18:59:47 UTC

Because it was talking about using overlayfs docker storage driver with disabled selinux inside docker daemon (via /etc/sysconfig/docker(-latest)) but with host's selinux ENABLED (getenforce in the host) which is supported, and atomic scan still errored out in the comment#8

On the other hand, I saw probably a different failure with atomic scan + overlay.
# atomic --debug scan --scanner openscap --scan_type cve registry.access.redhat.com/rhel7
Created /run/atomic/2016-10-04-18-51-55-243437
docker run -t --rm -v /etc/localtime:/etc/localtime -v /run/atomic/2016-10-04-18-51-55-243437:/scanin -v /var/lib/atomic/openscap/2016-10-04-18-51-55-243437:/scanout:rw,Z -v /etc/oscapd:/etc/oscapd:ro rhel7/openscap oscapd-evaluate scan --no-standard-compliance --targets chroots-in-dir:///scanin --output /scanout
Created /run/atomic/2016-10-04-18-51-55-243437/98a88a8b722a71835dd761c88451c681a8f1bc6e577f90d4dc8b234100bd4861
Mounted {u'Created': 1473186186, u'Labels': {u'distribution-scope': u'public', u'build-date': u'2016-09-06T14:12:54.553894Z', u'Vendor': u'Red Hat, Inc.', u'Name': u'rhel7/rhel', u'Build_Host': u'rcm-img-docker02.build.eng.bos.redhat.com', u'vcs-type': u'git', u'vcs-ref': u'08780b7a7779335cf28f64654e43c75ad9341c77', u'release': u'104', u'Version': u'7.2', u'Architecture': u'x86_64', u'Release': u'104', u'BZComponent': u'rhel-server-docker', u'Authoritative_Registry': u'registry.access.redhat.com', u'com.redhat.build-host': u'rcm-img-docker02.build.eng.bos.redhat.com', u'architecture': u'x86_64'}, 'ImageId': u'98a88a8b722a71835dd761c88451c681a8f1bc6e577f90d4dc8b234100bd4861', u'VirtualSize': 201376319, u'ParentId': u'', 'input': 'registry.access.redhat.com/rhel7', u'RepoTags': [u'registry.access.redhat.com/rhel7:latest'], u'RepoDigests': None, u'Id': u'98a88a8b722a71835dd761c88451c681a8f1bc6e577f90d4dc8b234100bd4861', 'ImageType': 'Docker', u'Size': 201376319} to /run/atomic/2016-10-04-18-51-55-243437/98a88a8b722a71835dd761c88451c681a8f1bc6e577f90d4dc8b234100bd4861
Creating the output dir at /var/lib/atomic/openscap/2016-10-04-18-51-55-243437
INFO:OpenSCAP Daemon one-off evaluator 0.1.5
INFO:Autodetected "oscap" in path "/usr/bin/oscap".
INFO:Autodetected "oscap-ssh" in path "/usr/bin/oscap-ssh".
INFO:Autodetected "oscap-vm" in path "/usr/bin/oscap-vm".
INFO:Autodetected "oscap-docker" in path "/usr/bin/oscap-docker".
INFO:Autodetected "oscap-chroot" in path "/usr/bin/oscap-chroot".
WARNING:Can't import the 'docker' package. Container scanning functionality will be disabled.
INFO:Autodetected SCAP content at "/usr/share/openscap/cpe/openscap-cpe-oval.xml".
INFO:Autodetected SCAP content in path "/usr/share/xml/scap/ssg/content".
INFO:Creating tasks directory at '/var/lib/oscapd/tasks' because it didn't exist.
INFO:Creating results directory at '/var/lib/oscapd/results' because it didn't exist.
INFO:Creating results work in progress directory at '/var/lib/oscapd/work_in_progress' because it didn't exist.
INFO:Evaluated EvaluationSpec, exit_code=0.
INFO:Had a local version of /var/lib/oscapd/cve_feeds/com.redhat.rhsa-RHEL7.xml but it wasn't new enough
INFO:Evaluated EvaluationSpec, exit_code=0.
INFO:[100.00%] Scanned target 'chroot:///scanin/98a88a8b722a71835dd761c88451c681a8f1bc6e577f90d4dc8b234100bd4861'

registry.access.redhat.com/rhel7 (98a88a8b722a718)

The following issues were found:

     RHSA-2016:1940: openssl security update (Important)
     Severity: Important
       RHSA URL: https://rhn.redhat.com/errata/RHSA-2016-1940.html
       RHSA ID: RHSA-2016:1940-01
       Associated CVEs:
           CVE ID: CVE-2016-2177
           CVE URL: https://access.redhat.com/security/cve/CVE-2016-2177
           CVE ID: CVE-2016-2178
           CVE URL: https://access.redhat.com/security/cve/CVE-2016-2178
           CVE ID: CVE-2016-2179
           CVE URL: https://access.redhat.com/security/cve/CVE-2016-2179
           CVE ID: CVE-2016-2180
           CVE URL: https://access.redhat.com/security/cve/CVE-2016-2180
           CVE ID: CVE-2016-2181
           CVE URL: https://access.redhat.com/security/cve/CVE-2016-2181
           CVE ID: CVE-2016-2182
           CVE URL: https://access.redhat.com/security/cve/CVE-2016-2182
           CVE ID: CVE-2016-6302
           CVE URL: https://access.redhat.com/security/cve/CVE-2016-6302
           CVE ID: CVE-2016-6304
           CVE URL: https://access.redhat.com/security/cve/CVE-2016-6304
           CVE ID: CVE-2016-6306
           CVE URL: https://access.redhat.com/security/cve/CVE-2016-6306


Files associated with this scan are in /var/lib/atomic/openscap/2016-10-04-18-51-55-243437.

Unmounted /run/atomic/2016-10-04-18-51-55-243437/98a88a8b722a71835dd761c88451c681a8f1bc6e577f90d4dc8b234100bd4861
g-io-error-quark: Refspec 'ociimage/9a011419912964fc07dca28c1276beee515c6d6546b1dc75cba05f6c350a6cbf-latest' not found (1)
Traceback (most recent call last):
  File "/bin/atomic", line 186, in <module>
    sys.exit(_func())
  File "/usr/lib/python2.7/site-packages/Atomic/scan.py", line 169, in scan
    self.record_environment()
  File "/usr/lib/python2.7/site-packages/Atomic/scan.py", line 395, in record_environment
    environment['images'].append(self._inspect_image(image=iid))
  File "/usr/lib/python2.7/site-packages/Atomic/atomic.py", line 254, in _inspect_image
    return self.syscontainers.inspect_system_image(image)
  File "/usr/lib/python2.7/site-packages/Atomic/syscontainers.py", line 607, in inspect_system_image
    return self._inspect_system_branch(repo, imagebranch)
  File "/usr/lib/python2.7/site-packages/Atomic/syscontainers.py", line 610, in _inspect_system_branch
    commit_rev = repo.resolve_rev(imagebranch, False)[1]
Error: g-io-error-quark: Refspec 'ociimage/9a011419912964fc07dca28c1276beee515c6d6546b1dc75cba05f6c350a6cbf-latest' not found (1)

Comment 12 Daniel Walsh 2016-10-04 19:14:10 UTC

Any avc messages?

Comment 13 Qian Cai 2016-10-04 19:17:28 UTC

No ausearch in atomic host and dmesg showed nothing. "setenforce 0" in the host did not help either. The trace looks like system container related, so I am not sure it might be just some bad interactive between atomic scan and system container.

Comment 14 Daniel Walsh 2016-10-04 19:21:15 UTC

Ok lets assign to Baude and see if he can recreate the error.

Comment 15 Qian Cai 2016-10-04 19:57:50 UTC

Confirmed the above can always be reproduced once a system container image existed and tracked here,
https://bugzilla.redhat.com/show_bug.cgi?id=1381717

Comment 17 errata-xmlrpc 2016-11-04 09:06:41 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2628.html

Note You need to log in before you can comment on or make changes to this bug.