Bug 1792423

Summary: toolbox fails to pull registry.redhat.io/rhel8/support-tools in an environment with HTTP/HTTPS Proxy
Product: OpenShift Container Platform Reporter: Olimp Bockowski <obockows>
Component: RHCOSAssignee: Allen Bai <abai>
Status: CLOSED ERRATA QA Contact: Michael Nguyen <mnguyen>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 4.2.0CC: andbartl, aos-bugs, bbreard, dornelas, dustymabe, imcleod, jligon, jokerman, miabbott, nstielau, pragshar, schoudha, scuppett, smilner
Target Milestone: ---   
Target Release: 4.4.0   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1798107 (view as bug list) Environment:
Last Closed: 2020-05-04 11:24:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1186913, 1798107, 1798109    

Description Olimp Bockowski 2020-01-17 16:09:04 UTC 
Description of problem:
toolbox fails to start due to: 

Error: error pulling image "registry.redhat.io/rhel8/support-tools": unable to pull registry.redhat.io/rhel8/support-tools: unable to pull image: Error initializing source docker://registry.redhat.io/rhel8/support-tools:latest: pinging docker registry returned: Get https://registry.redhat.io/v2/: dial tcp 104.103.100.204:443: i/o timeout

Version-Release number of selected component (if applicable):
OCP 4.x

How reproducible:
always

Steps to Reproduce:
1. set HTTP & HTTPS Proxy for an environment, no direct, external access
2. set HTTP_PROXY and HTTPS_PROXY accordingly
3. run toolbox

Actual results:
Fails, due to i/o timeout 

Expected results:
Downloads support-tools:latest image

Additional info:
The first detailed reason why it fails, then possible fixes. 
Env variables are exported, then toolbox is run. 
podman is here carthorse, I believe it uses net/http package and 
func ProxyFromEnvironment(req *Request) (*url.URL, error)
to read proxy settings (if needed)

toolbox checks is there an existing image: : 
 22     if ! image_exists; then
 23         image_pull
so here is a possible workaround I describe in KCS is to manually: podman login, podman pull)
Later "image_pull" tries using /var/lib/kubelet/config.json or manual login:
74     if ! sudo podman pull --authfile /var/lib/kubelet/config.json "$TOOLBOX_IMAGE"; then
...
 78             sudo podman login "${REGISTRY}"

the buggy lines are 74 and 78, we have here sudo that doesn't preserve environment variables. Envs are passed to sudo itself: 

[root@master0 ~] ps -ef | grep -i proxy
root     3829008 3823754  0 11:21 pts/1    00:00:00 sudo podman login registry.redhat.io
root     3829011 3829008  0 11:21 pts/1    00:00:00 podman login registry.redhat.io
[root@master0 ~]# cat /proc/3829008/environ | tr '\0' '\n' | grep -i proxy
HTTP_PROXY=http://212.31.70.50:8080
HTTPS_PROXY=http://212.31.70.50:8080
[root@master0 ~]# cat /proc/3829011/environ | tr '\0' '\n' | grep -i proxy

it should be like that:

[ obockows.csb  ~ ] $ diff --unified=0 ./toolbox ./toolbox-fixed 
--- ./toolbox	2020-01-17 16:48:14.368579602 +0100
+++ ./toolbox-fixed	2020-01-17 16:56:44.804600337 +0100
@@ -74 +74 @@
-    if ! sudo podman pull --authfile /var/lib/kubelet/config.json "$TOOLBOX_IMAGE"; then
+    if ! sudo --preserve-env podman pull --authfile /var/lib/kubelet/config.json "$TOOLBOX_IMAGE"; then
@@ -78,2 +78,2 @@
-            sudo podman login "${REGISTRY}"
-            sudo podman pull "$TOOLBOX_IMAGE"
+            sudo --preserve-env podman login "${REGISTRY}"
+            sudo --preserve-env podman pull "$TOOLBOX_IMAGE"

if not GNU-style, then just "-E"
Comment 1 Stephen Cuppett 2020-01-17 18:10:06 UTC 
Setting target release to the active development branch (4.4). For fixes, if any, which require backport to prior versions, clones of this BZ will be created.
Comment 7 Michael Nguyen 2020-02-05 19:17:28 UTC 
Verified on RHCOS 44.81.202002040001-0

[core@localhost ~]$ rpm-ostree status
State: idle
AutomaticUpdates: disabled
Deployments:
* ostree://07231478c2113678fbbd1a1d27dd8863de70c11fd9af11d7547896bfd0dd76c5
                   Version: 44.81.202002040001-0 (2020-02-04T00:06:58Z)

[core@localhost cache]$ env | grep proxy
https_proxy=http://192.168.122.31:3128
http_proxy=http://192.168.122.31:3128
[core@localhost cache]$ toolbox
Trying to pull registry.redhat.io/rhel8/support-tools...
  unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication
Error: error pulling image "registry.redhat.io/rhel8/support-tools": unable to pull registry.redhat.io/rhel8/support-tools: unable to pull image: Error initializing source docker://registry.redhat.io/rhel8/support-tools:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication
Would you like to manually authenticate to registry: 'registry.redhat.io' and try again? [y/N] y
Username: mnguyen
Password: 
Login Succeeded!
Trying to pull registry.redhat.io/rhel8/support-tools...
Getting image source signatures
Copying blob 0a4a43613721 done
Copying blob ff6f434a470a done
Copying blob eae5d284042d done
Copying config 53d1e01dae done
Writing manifest to image destination
Storing signatures
53d1e01dae0c44c45f36e72d2d1f0fa91069c147bbd9d2971335ecf2ca93b446
Spawning a container 'toolbox-core' with image 'registry.redhat.io/rhel8/support-tools'
Detected RUN label in the container image. Using that as the default...
command: podman run -it --name toolbox-core --privileged --ipc=host --net=host --pid=host -e HOST=/host -e NAME=toolbox-core -e IMAGE=registry.redhat.io/rhel8/support-tools:latest -v /run:/run -v /var/log:/var/log -v /etc/machine-id:/etc/machine-id -v /etc/localtime:/etc/localtime -v /:/host registry.redhat.io/rhel8/support-tools:latest
[root@localhost /]# sosreport 

sosreport (version 3.7)

This command will collect diagnostic and configuration information from
this Red Hat Enterprise Linux system and installed applications.

An archive containing the collected information will be generated in
/host/var/tmp/sos.w9jmi15o and may be provided to a Red Hat support
representative.

Any information provided to Red Hat will be treated in accordance with
the published support policies at:

  https://access.redhat.com/support/

The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.

No changes will be made to system configuration.

Press ENTER to continue, or CTRL-C to quit.
Comment 9 errata-xmlrpc 2020-05-04 11:24:53 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581