Description of problem: When deploying a cluster using HTTP Proxy the hosts are discovered but the installation fails when hosts enter stage "Writing image to disk" with error from hosts: "Host master-0-0: updated status from "installing-in-progress" to "error" (Failed - failed after 3 attempts, last error: failed executing nsenter [-t 1 -m -i -- coreos-installer install --image-url https://releases-art-rhcos.svc.ci.openshift.org/art/storage/releases/rhcos-4.6/46.82.202008260918-0/x86_64/rhcos-46.82.202008260918-0-metal.x86_64.raw.gz --insecure -i /opt/install-dir/master.ign /dev/sda] , error exit status 1. Output Error: fetching image URL Caused by: error sending request for url (https://releases-art-rhcos.svc.ci.openshift.org/art/storage/releases/rhcos-4.6/46.82.202008260918-0/x86_64/rhcos-46.82.202008260918-0-metal.x86_64.raw.gz): error trying to connect: tcp connect error: Network is unreachable (os error 101) Caused by: error trying to connect: tcp connect error: Network is unreachable (os error 101) Caused by: tcp connect error: Network is unreachable (os error 101) Caused by: Network is unreachable (os error 101))" Host Environment: VMs with with all FORWARD traffic rejected on hypervisor and squid proxy running on hypervisor. Version-Release number of selected component (if applicable): Staging: { "Username": "nshidlin", "CurrentApp": "Clusters", "ApplicationPath": "/openshift/assisted-installer/clusters/db9daf23-8b9d-4fcb-a036-a36c7de8cb53", "apps": [ { "name": "Chrome", "path": "apps/chrome/app.info.json", "version": "8c3ed2ee756a054cfac24c7934bcc88a3ae3c8ee" }, { "name": "Dashboard", "path": "apps/dashboard/app.info.json", "version": "12610ba3b722a2bba51124295802a40b03bfa059" }, { "name": "Inventory", "path": "apps/inventory/app.info.json", "version": "a45b358582697a8d98f00e52fd775645896c3301" }, { "name": "Remediations", "path": "apps/remediations/app.info.json", "version": "1de53dd4fde78fecae0693f46928f1cb4e397447" }, { "name": "Vulnerability", "path": "apps/vulnerability/app.info.json", "version": "ef1fc91cb32ce6e574616f97a055948aaa5643fe" }, { "name": "Compliance", "path": "apps/compliance/app.info.json", "version": "f99d6c5a0abd538054f1f63b60333a46415827f9" }, { "name": "Cost Management", "path": "apps/cost-management/app.info.json", "version": "f2260b4db54da4bdb8aa5d507e8d5e232000fc5f" }, { "name": "Advisor", "path": "apps/advisor/app.info.json", "version": "aa3f015bfa5b8d1ed8cd177c20b6bc29fa47fd70" }, { "name": "Drift", "path": "apps/drift/app.info.json", "version": "9a4e02d2bdf857ab201145a3f8467efa3bd75f15" }, { "name": "Migration Analytics", "path": "apps/migration-analytics/app.info.json", "version": "7f37fa8e9fa0d29d8b1d7be8ba4cfe22979cdaff" }, { "name": "Automation Hub", "path": "apps/automation-hub/app.info.json", "version": "5884daee9a75b899e0788324ab9cf61ede96af08" }, { "name": "Automation Analytics", "path": "apps/automation-analytics/app.info.json", "version": "89f9692730b63bf77849b61bfd25950e68a21678" }, { "name": "Policies", "path": "apps/policies/app.info.json", "version": "e928d3b90fde56f16cb008f06f9f54c9167b94ff" }, { "name": "Patch", "path": "apps/patch/app.info.json", "version": "1ab627ad8f5bd595ad851106502c8c4a8de80df8" }, { "name": "Automation Services Catalog", "path": "apps/catalog/app.info.json", "version": "98fb9e1bcca8681282b2e2f1c84391a4e454c1b9" }, { "name": "Approval", "path": "apps/approval/app.info.json", "version": "76dbd9fc2ba4fbf99d00282bb48bf20c38cacf33" }, { "name": "Sources", "path": "apps/sources/app.info.json", "version": "4e4c53ac7a93e2c12e64b2bd86b1b09ad7192790" } ] } assisted installer component versions: { "versions": { "assisted-ignition-generator": "quay.io/ocpmetal/assisted-ignition-generator:v1.0.9.1", "assisted-installer": "quay.io/ocpmetal/assisted-installer:v1.0.9.1", "assisted-installer-controller": "quay.io/ocpmetal/assisted-installer-controller:v1.0.9.1", "assisted-installer-service": "quay.io/app-sre/assisted-service:f0172a8", "discovery-agent": "quay.io/ocpmetal/assisted-installer-agent:v1.0.9.1", "image-builder": "quay.io/app-sre/assisted-iso-create:f0172a8" } } How reproducible: Evert time Steps to Reproduce: 1. Set HTTP proxy to URL:port of squid proxy when generating ISO 2. Download ISO and boot nodes 3. Once nodes are discovered complete cluster info and start installation Actual results: Hosts fail when they reach stage "Writing image to disk" Expected results: The cluster should complete installation Additional info:
The PR https://github.com/openshift/assisted-service/pull/287 was merged 3 days ago, but it didn't make to "assisted-installer-service": "quay.io/app-sre/assisted-service:f0172a8" Moving to ON_QE so this one can be verified on the next build.
Retested on staging: { "release_tag": "v1.0.9.2-ds", "versions": { "assisted-ignition-generator": "quay.io/ocpmetal/assisted-ignition-generator:v1.0.9.2", "assisted-installer": "registry-proxy.engineering.redhat.com/rh-osbs/openshift4-assisted-installer-rhel8:v4.6.0-14", "assisted-installer-controller": "registry-proxy.engineering.redhat.com/rh-osbs/openshift4-assisted-installer-reporter-rhel8:v4.6.0-10", "assisted-installer-service": "quay.io/app-sre/assisted-service:990e3e0", "discovery-agent": "registry-proxy.engineering.redhat.com/rh-osbs/openshift4-assisted-installer-agent-rhel8:v4.6.0-12", "image-builder": "quay.io/app-sre/assisted-iso-create:990e3e0" } } And the issue persists
This issue happens for d/s image of assisted-installer based on ubi8-minimal:8-released https://code.engineering.redhat.com/gerrit/gitweb?p=assisted-installer.git;a=blob;f=distgit/containers/assisted-installer/Dockerfile.in;h=640fe2a2c87d8653b0cbec48e3a680806a2d6c79;hb=refs/heads/rhaos-4.6-rhel-8 While U/S assisted-installer image is based on bash, was able to download the resource via the proxy settings. https://github.com/openshift/assisted-installer/blob/800610175080981aa67cb3a306bac0be667f8a89/Dockerfile.assisted-installer In order to make sure behavior is expected and aligned, proxy env vars will be provided explicitly to the commands being executed on the assisted-installer image on a process-id other than the agent. https://github.com/openshift/assisted-installer/pull/35 https://github.com/openshift/assisted-service/pull/362
Verified on staging 9/18: { "release_tag": "v1.0.9.4-ds", "versions": { "assisted-ignition-generator": "quay.io/ocpmetal/assisted-ignition-generator:v1.0.9.4", "assisted-installer": "registry-proxy.engineering.redhat.com/rh-osbs/openshift4-assisted-installer-rhel8:v4.6.0-19", "assisted-installer-controller": "registry-proxy.engineering.redhat.com/rh-osbs/openshift4-assisted-installer-reporter-rhel8:v4.6.0-15", "assisted-installer-service": "quay.io/app-sre/assisted-service:b793c52", "discovery-agent": "registry-proxy.engineering.redhat.com/rh-osbs/openshift4-assisted-installer-agent-rhel8:v4.6.0-15", "image-builder": "quay.io/app-sre/assisted-iso-create:b793c52" } }