Bug 1311544 - atomic scan tracebacks on scanning all images
atomic scan tracebacks on scanning all images
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: atomic (Show other bugs)
7.2
x86_64 Linux
unspecified Severity unspecified
: rc
: ---
Assigned To: Brent Baude
atomic-bugs@redhat.com
:
Depends On: 1343944 1343939 1346942
Blocks: 1296594 1313485
  Show dependency treegraph
 
Reported: 2016-02-24 07:39 EST by Jan Černý
Modified: 2016-06-23 12:21 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-06-23 12:21:23 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Jan Černý 2016-02-24 07:39:56 EST
Description of problem:

I use atomic scan command to scan containers and images for vulnerabilities.
I scan using super privileged container which is built with Dockerfile from here:
https://github.com/OpenSCAP/openscap-daemon/tree/master/atomic/rhel7_spc
I can scan of a single image or container successfully.
However, when I try to scan all images using
atomic scan --images or atomic scan --all,
the command crashes and I got a traceback.

Version-Release number of selected component (if applicable):

atomic-1.6-6.gitca1e384.el7.x86_64

How reproducible:

always

Steps to Reproduce:

1. Build, run and install RHEL7 SPC as described on https://github.com/OpenSCAP/openscap-daemon/blob/master/atomic/
2. atomic scan --images
3. see the traceback

Actual results:

[root@localhost ~]# atomic scan --images

Scanning...

Traceback (most recent call last):
  File "/usr/bin/atomic", line 420, in <module>
    sys.exit(args.func())
  File "/usr/lib/python2.7/site-packages/Atomic/atomic.py", line 485, in scan
    clean = util.print_scan_summary(scan_return, input_resolve)
  File "/usr/lib/python2.7/site-packages/Atomic/util.py", line 124, in print_scan_summary
    template = "{0:" + str(max_width) + "}   {1:5} {2:5} {3:5} {4:5}"
UnboundLocalError: local variable 'max_width' referenced before assignment


Expected results:

Displayed results of CVE scan of all images. No traceback.


Additional info:
Comment 2 Jan Černý 2016-02-24 09:43:59 EST
Applying this patch: https://github.com/projectatomic/atomic/commit/5fd44bbb58bbc86213bd726563d7bdc554d73cfa to /usr/lib/python2.7/site-packages/Atomic/util.py fixes the traceback.

I spent some time playing with atomic scan, scanning multiple images and containers and it worked OK for me with that patch applied.
Comment 3 Alex Jia 2016-02-25 03:10:01 EST
(In reply to Jan Černý from comment #0)

> [root@localhost ~]# atomic scan --images
> 
> Scanning...
> 
> Traceback (most recent call last):
>   File "/usr/bin/atomic", line 420, in <module>
>     sys.exit(args.func())
>   File "/usr/lib/python2.7/site-packages/Atomic/atomic.py", line 485, in scan
>     clean = util.print_scan_summary(scan_return, input_resolve)
>   File "/usr/lib/python2.7/site-packages/Atomic/util.py", line 124, in
> print_scan_summary
>     template = "{0:" + str(max_width) + "}   {1:5} {2:5} {3:5} {4:5}"
> UnboundLocalError: local variable 'max_width' referenced before assignment
> 

I have ever given a test for atomic-1.6-3.gitea18c14.el7.x86_64 w/ docker-1.9.0-8.el7.x86_64, it works well w/o above error, and also work on atomic-1.8-4.git958d939.el7.x86_64 w/ docker-1.9.1-12.el7.x86_64.
Comment 4 Jan Černý 2016-03-01 11:53:32 EST
(In reply to Alex Jia from comment #3)
> (In reply to Jan Černý from comment #0)
> 
> > [root@localhost ~]# atomic scan --images
> > 
> > Scanning...
> > 
> > Traceback (most recent call last):
> >   File "/usr/bin/atomic", line 420, in <module>
> >     sys.exit(args.func())
> >   File "/usr/lib/python2.7/site-packages/Atomic/atomic.py", line 485, in scan
> >     clean = util.print_scan_summary(scan_return, input_resolve)
> >   File "/usr/lib/python2.7/site-packages/Atomic/util.py", line 124, in
> > print_scan_summary
> >     template = "{0:" + str(max_width) + "}   {1:5} {2:5} {3:5} {4:5}"
> > UnboundLocalError: local variable 'max_width' referenced before assignment
> > 
> 
> I have ever given a test for atomic-1.6-3.gitea18c14.el7.x86_64 w/
> docker-1.9.0-8.el7.x86_64, it works well w/o above error, and also work on
> atomic-1.8-4.git958d939.el7.x86_64 w/ docker-1.9.1-12.el7.x86_64.

Why have you tested these particular versions of atomic? My bug occurs on 
atomic-1.6-6.gitca1e384.el7.x86_64. Which version is the most recent version available in RHEL 7.2 ?

I had a look into the code in Atomic/util.py, I can see that the variable max_width is undefined if len(names) == 0, so the traceback is inevitable.

Did you test scanning without specifying image ids or names?
eg. # atomic scan --all
Comment 5 Alex Jia 2016-03-01 22:10:08 EST
(In reply to Jan Černý from comment #4)
> > I have ever given a test for atomic-1.6-3.gitea18c14.el7.x86_64 w/
> > docker-1.9.0-8.el7.x86_64, it works well w/o above error, and also work on
> > atomic-1.8-4.git958d939.el7.x86_64 w/ docker-1.9.1-12.el7.x86_64.
> 
> Why have you tested these particular versions of atomic? My bug occurs on 
> atomic-1.6-6.gitca1e384.el7.x86_64. Which version is the most recent version
> available in RHEL 7.2 ?
> 

The atomic-1.6-3.gitea18c14.el7.x86_64 is a fixed version for bug 1272037, and the atomic-1.8-4.git958d939.el7.x86_64 is a errata version in that time. But anyway, you're right, I also should double check recent atomic version in RHEL 7.2.

> I had a look into the code in Atomic/util.py, I can see that the variable
> max_width is undefined if len(names) == 0, so the traceback is inevitable.
> 

Yes, I also noticed this, it's a python syntax error.

> Did you test scanning without specifying image ids or names?
> eg. # atomic scan --all

Yes, I indeed done this, for details, please see  Comment 24 in bug 1272037.
Comment 6 Daniel Walsh 2016-06-03 15:55:58 EDT
Fixed in atomic-1.10
Comment 11 Alex Jia 2016-06-16 05:31:39 EDT
Well, rhel7/openscap image is managed by atomic, so we need to run atomic install and run to execute INSTALL and RUN method firstly.

The current question is atomic scan will automatically pull rhel7/openscap image from repo if it doesn't exist on the local, but atomic scan hasn't execute atomic install and run method automatically.

It will be nice if we can help complete atomic install and run together, or if we don't wanna do this, at least, we should document this for users, and we should let users to pull rhel7/openscap image rather than automatically pull it by atomic scan.
Comment 14 Alex Jia 2016-06-16 11:01:52 EDT
Moving the bug to VERIFIED status per above workarround.
Comment 15 Daniel Walsh 2016-06-16 16:37:53 EDT
A workaround that set setenforce 0 is not a work around.  Is this testing using the unlock/overlayfs?  SELinux will not work with that, If this is fully up2date system with the latest packages, can you attach the avc's that you are seeing?  If this works on a RHEL system it should work on a RHELAH system in enforcing mode.
Comment 16 Alex Jia 2016-06-16 23:52:02 EDT
(In reply to Daniel Walsh from comment #15)
> A workaround that set setenforce 0 is not a work around.  Is this testing
> using the unlock/overlayfs?  SELinux will not work with that, If this is
> fully up2date system with the latest packages, can you attach the avc's that
> you are seeing?  If this works on a RHEL system it should work on a RHELAH
> system in enforcing mode.

Daniel, yes, because the current rhelah 7.2.5(5b82b4035f) hasn't included atomic-1.10.5-5, and it's failed[1] to upgrade Atomic Host since yesterday, in order to continue to push the testing forward, so I use atomic host unlock to install latest atomic on Atomic Host, for details, please see Comment 12. 

Daniel, need I wait for new rhelah 7.2.5 build(ship atomic-1.10.5-5) then give a try again? BTW, it indeed works on RHEL7 system w/ SELinux enforcing mode.


[1] error: Server returned status 500: Internal Server Error

[cloud-user@atomic-host-001 ~]$ sudo atomic host status
  TIMESTAMP (UTC)         VERSION     ID             OSNAME               REFSPEC                                                   
* 2016-06-13 17:33:11     7.2.5       5b82b4035f     rhel-atomic-host     rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard
  2016-05-06 05:57:30     7.2.4       b060975ce3     rhel-atomic-host     rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard
[cloud-user@atomic-host-001 ~]$ rpm -q atomic docker
atomic-1.10.5-3.el7.x86_64
docker-1.10.3-40.el7.x86_64
Comment 18 Daniel Walsh 2016-06-17 09:58:36 EDT
So you are not seeing the devicemapper problem?
Comment 19 Alex Jia 2016-06-17 12:48:13 EDT
(In reply to Daniel Walsh from comment #18)
> So you are not seeing the devicemapper problem?

Daniel, I still can see devicemapper problem[1], but atomic scanner can get scanning report w/o original issue in Comment 0.

[1] Error deleting container: Error response from daemon: Driver devicemapper failed to remove root filesystem a6c92e1b9b13e3351752b3b92b8dc14bf2ffeda8f34c777a3e1ce57d1854223d: remove /var/lib/docker/devicemapper/mnt/3680338ab00c3f2a4a6b3f2a74cc1bde6de53fd13f922b727da4f4d13358dff0: device or resource busy

The issue is tracked in bug 1347037.

500 Server Error: Internal Server Error ("devmapper: Unknown device 3797efd119f0691a9b73c868fecc1f29649dc3d7d2109830c6f562f5ac69fd40")

Sometimes, I can see above issue.
Comment 20 Daniel Walsh 2016-06-17 15:17:09 EDT
Alex add 

MountFlags=slave

to docker.service file and then restart docker daemon
Then try the scan again.

This is the fix we are working on now.
Comment 21 Alex Jia 2016-06-18 06:24:08 EDT
(In reply to Daniel Walsh from comment #20)
> Alex add 
> 
> MountFlags=slave
> 
> to docker.service file and then restart docker daemon
> Then try the scan again.
> 
> This is the fix we are working on now.

Daniel, thanks, I gave a try, but I can't use cloud-user account to edit /usr/lib/systemd/system/docker.service, even though using sudo.


[cloud-user@atomic-host-001 ~]$ sudo ls -lZ /usr/lib/systemd/system/docker.service
-rw-r--r--. root root system_u:object_r:docker_unit_file_t:s0 /usr/lib/systemd/system/docker.service

I will give a try w/ root user on rhelah or RHEL7 system later today.
Comment 22 Alex Jia 2016-06-18 11:01:34 EDT
Daniel, the root user also can't edit /usr/lib/systemd/system/docker.service in rhelah 7.2.5.

-bash-4.2# atomic host status
  TIMESTAMP (UTC)         VERSION     ID             OSNAME               REFSPEC                                                   
* 2016-06-15 21:08:10     7.2.5       c6530479e2     rhel-atomic-host     rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard
  2016-04-12 14:20:45     7.2.3-1     644fcc6035     rhel-atomic-host     rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard

-bash-4.2# id
uid=0(root) gid=0(root) groups=0(root) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
-bash-4.2# ll -Z /usr/lib/systemd/system/docker.service
-rw-r--r--. root root system_u:object_r:docker_unit_file_t:s0 /usr/lib/systemd/system/docker.service

And it seems MountFlags=slave works in my rhel7 system, I haven't see devicemapper errors, but I still see many mount points haven't been unmounted.

<slice>

/dev/dm-77             10474496  1030740   9443756  10% /run/atomic/2016-06-18-10-55-05-624109/sha256:a05ba1bc496e2bfb07ecae7ebabad56173da3b317576eea7cd00caf366352469
/dev/dm-78             10474496   228808  10245688   3% /run/atomic/2016-06-18-10-55-05-624109/sha256:bdfeab864e127ee6c87368d98fa1338b673e408254be52e22e8730b48e7ac499

</slice>
Comment 23 Daniel Walsh 2016-06-20 14:35:28 EDT
What I did to test this was to create a copy of docker.service to /etc/systemd/system/docker1.service

Add the mount line to docker1.service
systemctl stop docker
systemctl start docker1
Comment 24 Alex Jia 2016-06-22 03:00:56 EDT
(In reply to Daniel Walsh from comment #23)
> What I did to test this was to create a copy of docker.service to
> /etc/systemd/system/docker1.service

Daniel, it works well for me, need I open a bug to track the issue? or fix it with bug 1347037 together?
Comment 26 errata-xmlrpc 2016-06-23 12:21:23 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1273

Note You need to log in before you can comment on or make changes to this bug.