Bug 1974959

Summary:	[GSS] OCS must-gather fails in an airgapped environment
Product:	[Red Hat Storage] Red Hat OpenShift Container Storage	Reporter:	Andy Bartlett <andbartl>
Component:	must-gather	Assignee:	Gobinda Das <godas>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Daniel Horák <dahorak>
Severity:	high	Docs Contact:
Priority:	high
Version:	4.6	CC:	bniver, branto, gmeno, kjosy, mduasope, muagarwa, nberry, ndevos, ocs-bugs, rcyriac, sabose, shan, sostapov, tdesala
Target Milestone:	---
Target Release:	OCS 4.8.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	4.8.0-432.ci	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-02-09 17:29:02 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Andy Bartlett 2021-06-22 20:16:57 UTC

Description of problem (please be detailed as possible and provide log
snippests):

I have a customer who was trying to get an ocs must-gather in an airgapped environment, although they had mirrored the must-gather image across and run:

$ oc adm must-gather --image=customer_registry/ocs4/ocs-must-gather-rhel8:v4.6 --dest-dir=<directory-name>

This then fails:

POD 2021-06-22T16:20:05.939476441Z --2021-06-22 16:20:05--  http://stedolan.github.io/jq/download/linux64/jq
POD 2021-06-22T16:20:05.939476441Z Resolving stedolan.github.io (stedolan.github.io)... 2021-06-22T16:20:05.942399279Z 185.199.109.153, 185.199.108.153, 185.199.110.153, ...
POD 2021-06-22T16:20:05.942399279Z Connecting to stedolan.github.io (stedolan.github.io)|185.199.109.153|:80... 2021-06-22T16:22:05.835842881Z mv: cannot stat 'jq': No such file or directory

Version of all relevant components (if applicable):

OCP 4.6.27


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?


Is there any workaround available to the best of your knowledge?


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?


Can this issue reproducible?

Yes


Can this issue reproduce from the UI?

No


If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1.
2.
3.


Actual results:


Expected results:


Additional info:

This issue is caused by the image reaching out for jq from github which wont work in an airgapped cluster

Comment 2 Mudit Agarwal 2021-06-23 03:56:33 UTC

We already have an upstream issue raised for this problem.

Comment 5 Sébastien Han 2021-06-25 08:02:31 UTC

Upstream toolbox has the package already, so it's a downstream request to add the `jq` binary to the rook operator image.
Boris, can you help with this? Thanks

Comment 11 Dan Horák 2021-06-30 12:06:45 UTC

The other Daniel H. (dahorak), please :-)

Comment 12 Daniel Horák 2021-07-01 12:10:11 UTC

Reproduced on OCP 4.6.35, ocs-operator.v4.6.5, must-gather image ocs4/ocs-must-gather-rhel8:v4.6:

  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  $ oc adm must-gather --image=${MIRROR-REGISTRY}:5000/ocs4/ocs-must-gather-rhel8:v4.6 --dest-dir=ocs_must_gather
  ...
  [must-gather-qn8xj] POD 2021-06-30T06:48:48.860486508Z --2021-06-30 06:48:48--  http://stedolan.github.io/jq/download/linux64/jq
  [must-gather-qn8xj] POD 2021-06-30T06:48:48.860486508Z Resolving stedolan.github.io (stedolan.github.io)... 2021-06-30T06:48:48.866904321Z 185.199.110.153, 185.199.111.153, 185.199.109.153, ...
  [must-gather-qn8xj] POD 2021-06-30T06:48:48.866904321Z Connecting to stedolan.github.io (stedolan.github.io)|185.199.110.153|:80... 2021-06-30T06:50:48.866655828Z mv: cannot stat 'jq': No such file or directory
  [must-gather-qn8xj] POD 2021-06-30T06:50:48.868116656Z command terminated with exit code 1
  ...
  [must-gather-qn8xj] POD 2021-06-30T06:52:01.493005660Z bash: jq: command not found
  [must-gather-qn8xj] POD 2021-06-30T06:52:01.799734813Z Traceback (most recent call last):
  [must-gather-qn8xj] POD 2021-06-30T06:52:01.799734813Z   File "/usr/bin/ceph", line 1269, in <module>
  [must-gather-qn8xj] POD 2021-06-30T06:52:01.799734813Z     retval = main()
  [must-gather-qn8xj] POD 2021-06-30T06:52:01.799734813Z   File "/usr/bin/ceph", line 1254, in main
  [must-gather-qn8xj] POD 2021-06-30T06:52:01.799734813Z     sys.stdout.flush()
  [must-gather-qn8xj] POD 2021-06-30T06:52:01.799734813Z BrokenPipeError: [Errno 32] Broken pipe
  [must-gather-qn8xj] POD 2021-06-30T06:52:01.799734813Z Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
  [must-gather-qn8xj] POD 2021-06-30T06:52:01.799734813Z BrokenPipeError: [Errno 32] Broken pipe
  [must-gather-qn8xj] POD 2021-06-30T06:52:01.813122798Z command terminated with exit code 127
  ...

  $ echo $?
  0
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


Verified on OCP 4.8.0-0.nightly-2021-06-29-033219, ocs-operator.v4.8.0-432.ci,
ocs must-gather imae:
  quay.io/rhceph-dev/ocs-must-gather@sha256:318f9fb2c5a4aa470e5f221c4a7a76045b6778b4c8895db71bd9225024169371

There is no attempt to download anything from `github.io`:
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  $ oc adm must-gather --image=ec2-18-116-118-181.us-east-2.compute.amazonaws.com:5000/rhceph-dev/ocs-must-gather:latest-4.8 --dest-dir=ocs_must_gather | grep -e github.io -e jq
  $ echo $?
  1
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

>> VERIFIED