Bug 1415213

Summary:	Smart State with OCP 3.4 image-inspector hangs in "Extracting image"
Product:	Red Hat CloudForms Management Engine	Reporter:	Eduardo Minguez <eminguez>
Component:	Providers	Assignee:	Erez Freiberger <efreiber>
Status:	CLOSED ERRATA	QA Contact:	Einat Pacifici <epacific>
Severity:	high	Docs Contact:
Priority:	high
Version:	5.7.0	CC:	agallego, cpelland, epacific, fsimonce, haowang, jfrey, jhardy, ldomb, mifiedle, ncatling, obarenbo, simaishi, vestival
Target Milestone:	GA	Keywords:	TestOnly
Target Release:	5.7.1	Flags:	epacific: automate_bug+
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:	container
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	Previously running the SmartState Analysis for Container Images on OpenShift Container Platform 3.4 led to the Image Inspector Pod being blocked on the "Extracting image" step. CloudForms will use the new Image Inspector image in order to resolve the issue and successfully complete the SmartState Analysis Task.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-02-08 17:59:07 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	Container Management	Target Upstream Version:

Description Eduardo Minguez 2017-01-20 15:04:08 UTC

Description of problem:
I've trying to setup a smart state demo with CF4.2 and OCP 3.4 and it seems that the image-inspector pod hangs while "Extracting image":

2017/01/20 06:16:03 Pulling image registry.access.redhat.com/openshift3/image-inspector:2.1
2017/01/20 06:16:03 Finished Downloading Image (0Kb downloaded)
2017/01/20 06:16:05 Extracting image registry.access.redhat.com/openshift3/image-inspector:2.1 to /var/tmp/image-inspector-794506101

And the related task times out.

Version-Release number of selected component (if applicable):
CF4.2 GA

How reproducible:
CF4.2 GA + OCP 3.4 GA

Steps to Reproduce:
1.Attach a OCP provider to CF
2.Try to run smart state
3.The task will fail

Actual results:
The image-inspector pod hangs while "Extracting image":

2017/01/20 06:16:03 Pulling image registry.access.redhat.com/openshift3/image-inspector:2.1
2017/01/20 06:16:03 Finished Downloading Image (0Kb downloaded)
2017/01/20 06:16:05 Extracting image registry.access.redhat.com/openshift3/image-inspector:2.1 to /var/tmp/image-inspector-794506101

And the related task times out.


Expected results:
Smart state happens and there are some reports in CF

Additional info:

I've tried running manually image-inspector process in one of the nodes, and it also hangs.

[root@node1 ~]# image-inspector -image registry.access.redhat.com/openshift3/image-inspector:2.1
2017/01/20 06:01:31 Pulling image registry.access.redhat.com/openshift3/image-inspector:2.1
2017/01/20 06:01:31 Finished Downloading Image (0Kb downloaded)
2017/01/20 06:01:33 Extracting image registry.access.redhat.com/openshift3/image-inspector:2.1 to /var/tmp/image-inspector-132423015
^C

The issue seems to be related with docker-1.12 and this upstream issue https://github.com/openshift/image-inspector/issues/31

With docker-1.10 it works fine:
[root@bastion ~]# docker version | grep -i version
 Version:         1.10.3
[...]
[root@bastion ~]# image-inspector -image registry.access.redhat.com/openshift3/image-inspector:2.1 -serve 0.0.0.0:8080
2017/01/20 06:40:47 Pulling image registry.access.redhat.com/openshift3/image-inspector:2.1
2017/01/20 06:40:57 Downloading Image (82144Kb downloaded)
2017/01/20 06:41:07 Downloading Image (82144Kb downloaded)
2017/01/20 06:41:14 Finished Downloading Image (82144Kb downloaded)
2017/01/20 06:41:15 Extracting image registry.access.redhat.com/openshift3/image-inspector:2.1 to /var/tmp/image-inspector-630810915
2017/01/20 06:41:31 !!!WARNING!!! It is insecure to serve the image content without changing
2017/01/20 06:41:31 root (--chroot). Absolute-path symlinks in the image can lead to disclose
2017/01/20 06:41:31 information of the hosting system.
2017/01/20 06:41:31 Serving image content /var/tmp/image-inspector-630810915 on webdav://0.0.0.0:8080/api/v1/content/
[root@bastion ~]# yum update docker
Loaded plugins: search-disabled-repos
[...]                    
Complete!
[root@bastion ~]# systemctl restart docker
[root@bastion ~]# docker version | grep -i version
 Version:         1.12.5
[...]
[root@bastion ~]# image-inspector -image registry.access.redhat.com/openshift3/image-inspector:2.1 -serve 0.0.0.0:8080
2017/01/20 06:43:37 Pulling image registry.access.redhat.com/openshift3/image-inspector:2.1
2017/01/20 06:43:37 Finished Downloading Image (0Kb downloaded)
2017/01/20 06:43:38 Extracting image registry.access.redhat.com/openshift3/image-inspector:2.1 to /var/tmp/image-inspector-715653278
<hangs>

Comment 3 Federico Simoncelli 2017-01-23 08:49:10 UTC

Fix is most likely:

https://github.com/openshift/image-inspector/issues/31

This will require a new image-inspector build.

Comment 10 Einat Pacifici 2017-02-01 13:19:37 UTC

Verified with new image of image-inspector. 
Image inspector is set to take image from:
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/image-inspector:2.1-7

From "oc describe" image-inspector:
  image-inspector:
   .....
.....
    Image:		brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/image-inspector:2.1-7
    Image ID:		docker-pullable://brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/image-inspector@sha256:e47aec4c2de6b9b29e0e75990ce0b831ae29cd84e7df8cc52e9519fc52597f82


When testing with the above. The image-inspector no longer hangs with "Extracting image" error.

Comment 14 Einat Pacifici 2017-02-06 19:50:34 UTC

Verified with the following metrics: 

- CFME 4.1 + OpenShift 3.3 - passed
- CFME 4.1 + OpenShift 3.4 - passed
- CFME 4.2 + OpenShift 3.3 - passed
- CFME 4.2 + OpenShift 3.4 - passed


Please note: 
1. (as stated in comment 10) that Image inspector is set to take image from:
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/image-inspector:2.1-7
2. Several random images were selected during regression testing to see that SSA completes successfully.

Comment 18 errata-xmlrpc 2017-02-08 17:59:07 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0262