Bug 1415213 - Smart State with OCP 3.4 image-inspector hangs in "Extracting image"
Summary: Smart State with OCP 3.4 image-inspector hangs in "Extracting image"
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Providers
Version: 5.7.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: GA
: 5.7.1
Assignee: Erez Freiberger
QA Contact: Einat Pacifici
URL:
Whiteboard: container
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-01-20 15:04 UTC by Eduardo Minguez
Modified: 2018-04-17 11:31 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Previously running the SmartState Analysis for Container Images on OpenShift Container Platform 3.4 led to the Image Inspector Pod being blocked on the "Extracting image" step. CloudForms will use the new Image Inspector image in order to resolve the issue and successfully complete the SmartState Analysis Task.
Clone Of:
Environment:
Last Closed: 2017-02-08 17:59:07 UTC
Category: ---
Cloudforms Team: Container Management
Target Upstream Version:
epacific: automate_bug+


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:0262 normal SHIPPED_LIVE Red Hat OpenShift Container Platform Image Inspector image bug fix update 2017-02-08 23:09:09 UTC

Description Eduardo Minguez 2017-01-20 15:04:08 UTC
Description of problem:
I've trying to setup a smart state demo with CF4.2 and OCP 3.4 and it seems that the image-inspector pod hangs while "Extracting image":

2017/01/20 06:16:03 Pulling image registry.access.redhat.com/openshift3/image-inspector:2.1
2017/01/20 06:16:03 Finished Downloading Image (0Kb downloaded)
2017/01/20 06:16:05 Extracting image registry.access.redhat.com/openshift3/image-inspector:2.1 to /var/tmp/image-inspector-794506101

And the related task times out.

Version-Release number of selected component (if applicable):
CF4.2 GA

How reproducible:
CF4.2 GA + OCP 3.4 GA

Steps to Reproduce:
1.Attach a OCP provider to CF
2.Try to run smart state
3.The task will fail

Actual results:
The image-inspector pod hangs while "Extracting image":

2017/01/20 06:16:03 Pulling image registry.access.redhat.com/openshift3/image-inspector:2.1
2017/01/20 06:16:03 Finished Downloading Image (0Kb downloaded)
2017/01/20 06:16:05 Extracting image registry.access.redhat.com/openshift3/image-inspector:2.1 to /var/tmp/image-inspector-794506101

And the related task times out.


Expected results:
Smart state happens and there are some reports in CF

Additional info:

I've tried running manually image-inspector process in one of the nodes, and it also hangs.

[root@node1 ~]# image-inspector -image registry.access.redhat.com/openshift3/image-inspector:2.1
2017/01/20 06:01:31 Pulling image registry.access.redhat.com/openshift3/image-inspector:2.1
2017/01/20 06:01:31 Finished Downloading Image (0Kb downloaded)
2017/01/20 06:01:33 Extracting image registry.access.redhat.com/openshift3/image-inspector:2.1 to /var/tmp/image-inspector-132423015
^C

The issue seems to be related with docker-1.12 and this upstream issue https://github.com/openshift/image-inspector/issues/31

With docker-1.10 it works fine:
[root@bastion ~]# docker version | grep -i version
 Version:         1.10.3
[...]
[root@bastion ~]# image-inspector -image registry.access.redhat.com/openshift3/image-inspector:2.1 -serve 0.0.0.0:8080
2017/01/20 06:40:47 Pulling image registry.access.redhat.com/openshift3/image-inspector:2.1
2017/01/20 06:40:57 Downloading Image (82144Kb downloaded)
2017/01/20 06:41:07 Downloading Image (82144Kb downloaded)
2017/01/20 06:41:14 Finished Downloading Image (82144Kb downloaded)
2017/01/20 06:41:15 Extracting image registry.access.redhat.com/openshift3/image-inspector:2.1 to /var/tmp/image-inspector-630810915
2017/01/20 06:41:31 !!!WARNING!!! It is insecure to serve the image content without changing
2017/01/20 06:41:31 root (--chroot). Absolute-path symlinks in the image can lead to disclose
2017/01/20 06:41:31 information of the hosting system.
2017/01/20 06:41:31 Serving image content /var/tmp/image-inspector-630810915 on webdav://0.0.0.0:8080/api/v1/content/
[root@bastion ~]# yum update docker
Loaded plugins: search-disabled-repos
[...]                    
Complete!
[root@bastion ~]# systemctl restart docker
[root@bastion ~]# docker version | grep -i version
 Version:         1.12.5
[...]
[root@bastion ~]# image-inspector -image registry.access.redhat.com/openshift3/image-inspector:2.1 -serve 0.0.0.0:8080
2017/01/20 06:43:37 Pulling image registry.access.redhat.com/openshift3/image-inspector:2.1
2017/01/20 06:43:37 Finished Downloading Image (0Kb downloaded)
2017/01/20 06:43:38 Extracting image registry.access.redhat.com/openshift3/image-inspector:2.1 to /var/tmp/image-inspector-715653278
<hangs>

Comment 3 Federico Simoncelli 2017-01-23 08:49:10 UTC
Fix is most likely:

https://github.com/openshift/image-inspector/issues/31

This will require a new image-inspector build.

Comment 10 Einat Pacifici 2017-02-01 13:19:37 UTC
Verified with new image of image-inspector. 
Image inspector is set to take image from:
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/image-inspector:2.1-7

From "oc describe" image-inspector:
  image-inspector:
   .....
.....
    Image:		brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/image-inspector:2.1-7
    Image ID:		docker-pullable://brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/image-inspector@sha256:e47aec4c2de6b9b29e0e75990ce0b831ae29cd84e7df8cc52e9519fc52597f82


When testing with the above. The image-inspector no longer hangs with "Extracting image" error.

Comment 14 Einat Pacifici 2017-02-06 19:50:34 UTC
Verified with the following metrics: 

- CFME 4.1 + OpenShift 3.3 - passed
- CFME 4.1 + OpenShift 3.4 - passed
- CFME 4.2 + OpenShift 3.3 - passed
- CFME 4.2 + OpenShift 3.4 - passed


Please note: 
1. (as stated in comment 10) that Image inspector is set to take image from:
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/image-inspector:2.1-7
2. Several random images were selected during regression testing to see that SSA completes successfully.

Comment 18 errata-xmlrpc 2017-02-08 17:59:07 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0262


Note You need to log in before you can comment on or make changes to this bug.