Bug 1974959 - [GSS] OCS must-gather fails in an airgapped environment
Summary: [GSS] OCS must-gather fails in an airgapped environment
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat OpenShift Container Storage
Classification: Red Hat Storage
Component: must-gather
Version: 4.6
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: OCS 4.8.0
Assignee: Gobinda Das
QA Contact: Daniel Horák
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-06-22 20:16 UTC by Andy Bartlett
Modified: 2022-02-09 17:29 UTC (History)
14 users (show)

Fixed In Version: 4.8.0-432.ci
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-02-09 17:29:02 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift ocs-operator issues 1187 0 None open must-gather pod fails to fetch jq in a disconnected/restricted network cluster 2021-06-23 03:56:31 UTC
Github openshift ocs-operator pull 1244 0 None open must-gather: Don't download and install jq 2021-06-28 10:32:28 UTC
Github openshift ocs-operator pull 1245 0 None open Bug 1974959: [release-4.8] must-gather: Don't download and install jq 2021-06-28 12:06:15 UTC
Github red-hat-storage ocs-ci pull 5330 0 None Merged Must-Gather, disconnected environment, bz-1974959 2022-01-19 07:32:48 UTC

Description Andy Bartlett 2021-06-22 20:16:57 UTC
Description of problem (please be detailed as possible and provide log
snippests):

I have a customer who was trying to get an ocs must-gather in an airgapped environment, although they had mirrored the must-gather image across and run:

$ oc adm must-gather --image=customer_registry/ocs4/ocs-must-gather-rhel8:v4.6 --dest-dir=<directory-name>

This then fails:

POD 2021-06-22T16:20:05.939476441Z --2021-06-22 16:20:05--  http://stedolan.github.io/jq/download/linux64/jq
POD 2021-06-22T16:20:05.939476441Z Resolving stedolan.github.io (stedolan.github.io)... 2021-06-22T16:20:05.942399279Z 185.199.109.153, 185.199.108.153, 185.199.110.153, ...
POD 2021-06-22T16:20:05.942399279Z Connecting to stedolan.github.io (stedolan.github.io)|185.199.109.153|:80... 2021-06-22T16:22:05.835842881Z mv: cannot stat 'jq': No such file or directory

Version of all relevant components (if applicable):

OCP 4.6.27


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?


Is there any workaround available to the best of your knowledge?


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?


Can this issue reproducible?

Yes


Can this issue reproduce from the UI?

No


If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1.
2.
3.


Actual results:


Expected results:


Additional info:

This issue is caused by the image reaching out for jq from github which wont work in an airgapped cluster

Comment 2 Mudit Agarwal 2021-06-23 03:56:33 UTC
We already have an upstream issue raised for this problem.

Comment 5 Sébastien Han 2021-06-25 08:02:31 UTC
Upstream toolbox has the package already, so it's a downstream request to add the `jq` binary to the rook operator image.
Boris, can you help with this? Thanks

Comment 11 Dan Horák 2021-06-30 12:06:45 UTC
The other Daniel H. (dahorak), please :-)

Comment 12 Daniel Horák 2021-07-01 12:10:11 UTC
Reproduced on OCP 4.6.35, ocs-operator.v4.6.5, must-gather image ocs4/ocs-must-gather-rhel8:v4.6:

  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  $ oc adm must-gather --image=${MIRROR-REGISTRY}:5000/ocs4/ocs-must-gather-rhel8:v4.6 --dest-dir=ocs_must_gather
  ...
  [must-gather-qn8xj] POD 2021-06-30T06:48:48.860486508Z --2021-06-30 06:48:48--  http://stedolan.github.io/jq/download/linux64/jq
  [must-gather-qn8xj] POD 2021-06-30T06:48:48.860486508Z Resolving stedolan.github.io (stedolan.github.io)... 2021-06-30T06:48:48.866904321Z 185.199.110.153, 185.199.111.153, 185.199.109.153, ...
  [must-gather-qn8xj] POD 2021-06-30T06:48:48.866904321Z Connecting to stedolan.github.io (stedolan.github.io)|185.199.110.153|:80... 2021-06-30T06:50:48.866655828Z mv: cannot stat 'jq': No such file or directory
  [must-gather-qn8xj] POD 2021-06-30T06:50:48.868116656Z command terminated with exit code 1
  ...
  [must-gather-qn8xj] POD 2021-06-30T06:52:01.493005660Z bash: jq: command not found
  [must-gather-qn8xj] POD 2021-06-30T06:52:01.799734813Z Traceback (most recent call last):
  [must-gather-qn8xj] POD 2021-06-30T06:52:01.799734813Z   File "/usr/bin/ceph", line 1269, in <module>
  [must-gather-qn8xj] POD 2021-06-30T06:52:01.799734813Z     retval = main()
  [must-gather-qn8xj] POD 2021-06-30T06:52:01.799734813Z   File "/usr/bin/ceph", line 1254, in main
  [must-gather-qn8xj] POD 2021-06-30T06:52:01.799734813Z     sys.stdout.flush()
  [must-gather-qn8xj] POD 2021-06-30T06:52:01.799734813Z BrokenPipeError: [Errno 32] Broken pipe
  [must-gather-qn8xj] POD 2021-06-30T06:52:01.799734813Z Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
  [must-gather-qn8xj] POD 2021-06-30T06:52:01.799734813Z BrokenPipeError: [Errno 32] Broken pipe
  [must-gather-qn8xj] POD 2021-06-30T06:52:01.813122798Z command terminated with exit code 127
  ...

  $ echo $?
  0
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


Verified on OCP 4.8.0-0.nightly-2021-06-29-033219, ocs-operator.v4.8.0-432.ci,
ocs must-gather imae:
  quay.io/rhceph-dev/ocs-must-gather@sha256:318f9fb2c5a4aa470e5f221c4a7a76045b6778b4c8895db71bd9225024169371

There is no attempt to download anything from `github.io`:
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  $ oc adm must-gather --image=ec2-18-116-118-181.us-east-2.compute.amazonaws.com:5000/rhceph-dev/ocs-must-gather:latest-4.8 --dest-dir=ocs_must_gather | grep -e github.io -e jq
  $ echo $?
  1
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

>> VERIFIED


Note You need to log in before you can comment on or make changes to this bug.