Hide Forgot
Description of problem: toolbox fails to start due to: Error: error pulling image "registry.redhat.io/rhel8/support-tools": unable to pull registry.redhat.io/rhel8/support-tools: unable to pull image: Error initializing source docker://registry.redhat.io/rhel8/support-tools:latest: pinging docker registry returned: Get https://registry.redhat.io/v2/: dial tcp 104.103.100.204:443: i/o timeout Version-Release number of selected component (if applicable): OCP 4.x How reproducible: always Steps to Reproduce: 1. set HTTP & HTTPS Proxy for an environment, no direct, external access 2. set HTTP_PROXY and HTTPS_PROXY accordingly 3. run toolbox Actual results: Fails, due to i/o timeout Expected results: Downloads support-tools:latest image Additional info: The first detailed reason why it fails, then possible fixes. Env variables are exported, then toolbox is run. podman is here carthorse, I believe it uses net/http package and func ProxyFromEnvironment(req *Request) (*url.URL, error) to read proxy settings (if needed) toolbox checks is there an existing image: : 22 if ! image_exists; then 23 image_pull so here is a possible workaround I describe in KCS is to manually: podman login, podman pull) Later "image_pull" tries using /var/lib/kubelet/config.json or manual login: 74 if ! sudo podman pull --authfile /var/lib/kubelet/config.json "$TOOLBOX_IMAGE"; then ... 78 sudo podman login "${REGISTRY}" the buggy lines are 74 and 78, we have here sudo that doesn't preserve environment variables. Envs are passed to sudo itself: [root@master0 ~] ps -ef | grep -i proxy root 3829008 3823754 0 11:21 pts/1 00:00:00 sudo podman login registry.redhat.io root 3829011 3829008 0 11:21 pts/1 00:00:00 podman login registry.redhat.io [root@master0 ~]# cat /proc/3829008/environ | tr '\0' '\n' | grep -i proxy HTTP_PROXY=http://212.31.70.50:8080 HTTPS_PROXY=http://212.31.70.50:8080 [root@master0 ~]# cat /proc/3829011/environ | tr '\0' '\n' | grep -i proxy it should be like that: [ obockows@obockows.brq.csb ~ ] $ diff --unified=0 ./toolbox ./toolbox-fixed --- ./toolbox 2020-01-17 16:48:14.368579602 +0100 +++ ./toolbox-fixed 2020-01-17 16:56:44.804600337 +0100 @@ -74 +74 @@ - if ! sudo podman pull --authfile /var/lib/kubelet/config.json "$TOOLBOX_IMAGE"; then + if ! sudo --preserve-env podman pull --authfile /var/lib/kubelet/config.json "$TOOLBOX_IMAGE"; then @@ -78,2 +78,2 @@ - sudo podman login "${REGISTRY}" - sudo podman pull "$TOOLBOX_IMAGE" + sudo --preserve-env podman login "${REGISTRY}" + sudo --preserve-env podman pull "$TOOLBOX_IMAGE" if not GNU-style, then just "-E"
Setting target release to the active development branch (4.4). For fixes, if any, which require backport to prior versions, clones of this BZ will be created.
Verified on RHCOS 44.81.202002040001-0 [core@localhost ~]$ rpm-ostree status State: idle AutomaticUpdates: disabled Deployments: * ostree://07231478c2113678fbbd1a1d27dd8863de70c11fd9af11d7547896bfd0dd76c5 Version: 44.81.202002040001-0 (2020-02-04T00:06:58Z) [core@localhost cache]$ env | grep proxy https_proxy=http://192.168.122.31:3128 http_proxy=http://192.168.122.31:3128 [core@localhost cache]$ toolbox Trying to pull registry.redhat.io/rhel8/support-tools... unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication Error: error pulling image "registry.redhat.io/rhel8/support-tools": unable to pull registry.redhat.io/rhel8/support-tools: unable to pull image: Error initializing source docker://registry.redhat.io/rhel8/support-tools:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication Would you like to manually authenticate to registry: 'registry.redhat.io' and try again? [y/N] y Username: mnguyen@redhat.com Password: Login Succeeded! Trying to pull registry.redhat.io/rhel8/support-tools... Getting image source signatures Copying blob 0a4a43613721 done Copying blob ff6f434a470a done Copying blob eae5d284042d done Copying config 53d1e01dae done Writing manifest to image destination Storing signatures 53d1e01dae0c44c45f36e72d2d1f0fa91069c147bbd9d2971335ecf2ca93b446 Spawning a container 'toolbox-core' with image 'registry.redhat.io/rhel8/support-tools' Detected RUN label in the container image. Using that as the default... command: podman run -it --name toolbox-core --privileged --ipc=host --net=host --pid=host -e HOST=/host -e NAME=toolbox-core -e IMAGE=registry.redhat.io/rhel8/support-tools:latest -v /run:/run -v /var/log:/var/log -v /etc/machine-id:/etc/machine-id -v /etc/localtime:/etc/localtime -v /:/host registry.redhat.io/rhel8/support-tools:latest [root@localhost /]# sosreport sosreport (version 3.7) This command will collect diagnostic and configuration information from this Red Hat Enterprise Linux system and installed applications. An archive containing the collected information will be generated in /host/var/tmp/sos.w9jmi15o and may be provided to a Red Hat support representative. Any information provided to Red Hat will be treated in accordance with the published support policies at: https://access.redhat.com/support/ The generated archive may contain data considered sensitive and its content should be reviewed by the originating organization before being passed to any third party. No changes will be made to system configuration. Press ENTER to continue, or CTRL-C to quit.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0581