Bug 1877757 - [Assisted-4.6 ]Hosts unable to pull image during installation when using HTTP proxy
Summary: [Assisted-4.6 ]Hosts unable to pull image during installation when using HTTP...
Keywords:
Status: VERIFIED
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: assisted-installer
Version: 4.6
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: ---
Assignee: Moti Asayag
QA Contact: Yuri Obshansky
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-09-10 11:44 UTC by nshidlin
Modified: 2021-01-08 08:17 UTC (History)
3 users (show)

Fixed In Version: OCP-Metal-v1.0.9.5
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift assisted-service pull 287 0 None closed Use proxy for discovery image 2021-02-02 23:09:13 UTC

Description nshidlin 2020-09-10 11:44:54 UTC
Description of problem:

When deploying a cluster using HTTP Proxy the hosts are discovered but the installation fails when hosts enter stage "Writing image to disk" with error from hosts:

"Host master-0-0: updated status from "installing-in-progress" to "error" (Failed - failed after 3 attempts, last error: failed executing nsenter [-t 1 -m -i -- coreos-installer install --image-url https://releases-art-rhcos.svc.ci.openshift.org/art/storage/releases/rhcos-4.6/46.82.202008260918-0/x86_64/rhcos-46.82.202008260918-0-metal.x86_64.raw.gz --insecure -i /opt/install-dir/master.ign /dev/sda] , error exit status 1. Output Error: fetching image URL Caused by: error sending request for url (https://releases-art-rhcos.svc.ci.openshift.org/art/storage/releases/rhcos-4.6/46.82.202008260918-0/x86_64/rhcos-46.82.202008260918-0-metal.x86_64.raw.gz): error trying to connect: tcp connect error: Network is unreachable (os error 101) Caused by: error trying to connect: tcp connect error: Network is unreachable (os error 101) Caused by: tcp connect error: Network is unreachable (os error 101) Caused by: Network is unreachable (os error 101))"

Host Environment:

VMs with with all FORWARD traffic rejected on hypervisor and squid proxy running on hypervisor.

Version-Release number of selected component (if applicable):
Staging:
{
  "Username": "nshidlin",
  "CurrentApp": "Clusters",
  "ApplicationPath": "/openshift/assisted-installer/clusters/db9daf23-8b9d-4fcb-a036-a36c7de8cb53",
  "apps": [
    {
      "name": "Chrome",
      "path": "apps/chrome/app.info.json",
      "version": "8c3ed2ee756a054cfac24c7934bcc88a3ae3c8ee"
    },
    {
      "name": "Dashboard",
      "path": "apps/dashboard/app.info.json",
      "version": "12610ba3b722a2bba51124295802a40b03bfa059"
    },
    {
      "name": "Inventory",
      "path": "apps/inventory/app.info.json",
      "version": "a45b358582697a8d98f00e52fd775645896c3301"
    },
    {
      "name": "Remediations",
      "path": "apps/remediations/app.info.json",
      "version": "1de53dd4fde78fecae0693f46928f1cb4e397447"
    },
    {
      "name": "Vulnerability",
      "path": "apps/vulnerability/app.info.json",
      "version": "ef1fc91cb32ce6e574616f97a055948aaa5643fe"
    },
    {
      "name": "Compliance",
      "path": "apps/compliance/app.info.json",
      "version": "f99d6c5a0abd538054f1f63b60333a46415827f9"
    },
    {
      "name": "Cost Management",
      "path": "apps/cost-management/app.info.json",
      "version": "f2260b4db54da4bdb8aa5d507e8d5e232000fc5f"
    },
    {
      "name": "Advisor",
      "path": "apps/advisor/app.info.json",
      "version": "aa3f015bfa5b8d1ed8cd177c20b6bc29fa47fd70"
    },
    {
      "name": "Drift",
      "path": "apps/drift/app.info.json",
      "version": "9a4e02d2bdf857ab201145a3f8467efa3bd75f15"
    },
    {
      "name": "Migration Analytics",
      "path": "apps/migration-analytics/app.info.json",
      "version": "7f37fa8e9fa0d29d8b1d7be8ba4cfe22979cdaff"
    },
    {
      "name": "Automation Hub",
      "path": "apps/automation-hub/app.info.json",
      "version": "5884daee9a75b899e0788324ab9cf61ede96af08"
    },
    {
      "name": "Automation Analytics",
      "path": "apps/automation-analytics/app.info.json",
      "version": "89f9692730b63bf77849b61bfd25950e68a21678"
    },
    {
      "name": "Policies",
      "path": "apps/policies/app.info.json",
      "version": "e928d3b90fde56f16cb008f06f9f54c9167b94ff"
    },
    {
      "name": "Patch",
      "path": "apps/patch/app.info.json",
      "version": "1ab627ad8f5bd595ad851106502c8c4a8de80df8"
    },
    {
      "name": "Automation Services Catalog",
      "path": "apps/catalog/app.info.json",
      "version": "98fb9e1bcca8681282b2e2f1c84391a4e454c1b9"
    },
    {
      "name": "Approval",
      "path": "apps/approval/app.info.json",
      "version": "76dbd9fc2ba4fbf99d00282bb48bf20c38cacf33"
    },
    {
      "name": "Sources",
      "path": "apps/sources/app.info.json",
      "version": "4e4c53ac7a93e2c12e64b2bd86b1b09ad7192790"
    }
  ]
}

assisted installer component versions:
{
    "versions": {
        "assisted-ignition-generator": "quay.io/ocpmetal/assisted-ignition-generator:v1.0.9.1",
        "assisted-installer": "quay.io/ocpmetal/assisted-installer:v1.0.9.1",
        "assisted-installer-controller": "quay.io/ocpmetal/assisted-installer-controller:v1.0.9.1",
        "assisted-installer-service": "quay.io/app-sre/assisted-service:f0172a8",
        "discovery-agent": "quay.io/ocpmetal/assisted-installer-agent:v1.0.9.1",
        "image-builder": "quay.io/app-sre/assisted-iso-create:f0172a8"
    }
}

How reproducible:
Evert time

Steps to Reproduce:
1. Set HTTP proxy to URL:port of squid proxy when generating ISO
2. Download ISO and boot nodes
3. Once nodes are discovered complete cluster info and start installation

Actual results:
Hosts fail when they reach stage "Writing image to disk"

Expected results:
The cluster should complete installation

Additional info:

Comment 1 Moti Asayag 2020-09-10 16:28:13 UTC
The PR https://github.com/openshift/assisted-service/pull/287 was merged 3 days ago, but it didn't make to "assisted-installer-service": "quay.io/app-sre/assisted-service:f0172a8"

Moving to ON_QE so this one can be verified on the next build.

Comment 2 nshidlin 2020-09-13 07:20:59 UTC
Retested on staging:
{
    "release_tag": "v1.0.9.2-ds",
    "versions": {
        "assisted-ignition-generator": "quay.io/ocpmetal/assisted-ignition-generator:v1.0.9.2",
        "assisted-installer": "registry-proxy.engineering.redhat.com/rh-osbs/openshift4-assisted-installer-rhel8:v4.6.0-14",
        "assisted-installer-controller": "registry-proxy.engineering.redhat.com/rh-osbs/openshift4-assisted-installer-reporter-rhel8:v4.6.0-10",
        "assisted-installer-service": "quay.io/app-sre/assisted-service:990e3e0",
        "discovery-agent": "registry-proxy.engineering.redhat.com/rh-osbs/openshift4-assisted-installer-agent-rhel8:v4.6.0-12",
        "image-builder": "quay.io/app-sre/assisted-iso-create:990e3e0"
    }
}

And the issue persists

Comment 3 Moti Asayag 2020-09-14 17:18:59 UTC
This issue happens for d/s image of assisted-installer based on ubi8-minimal:8-released
https://code.engineering.redhat.com/gerrit/gitweb?p=assisted-installer.git;a=blob;f=distgit/containers/assisted-installer/Dockerfile.in;h=640fe2a2c87d8653b0cbec48e3a680806a2d6c79;hb=refs/heads/rhaos-4.6-rhel-8

While U/S assisted-installer image is based on bash, was able to download the resource via the proxy settings.
https://github.com/openshift/assisted-installer/blob/800610175080981aa67cb3a306bac0be667f8a89/Dockerfile.assisted-installer

In order to make sure behavior is expected and aligned, proxy env vars will be provided explicitly to the commands being executed on the assisted-installer image on a process-id other than the agent.

https://github.com/openshift/assisted-installer/pull/35
https://github.com/openshift/assisted-service/pull/362

Comment 4 nshidlin 2020-09-18 12:18:22 UTC
Verified on staging 9/18:
{
    "release_tag": "v1.0.9.4-ds",
    "versions": {
        "assisted-ignition-generator": "quay.io/ocpmetal/assisted-ignition-generator:v1.0.9.4",
        "assisted-installer": "registry-proxy.engineering.redhat.com/rh-osbs/openshift4-assisted-installer-rhel8:v4.6.0-19",
        "assisted-installer-controller": "registry-proxy.engineering.redhat.com/rh-osbs/openshift4-assisted-installer-reporter-rhel8:v4.6.0-15",
        "assisted-installer-service": "quay.io/app-sre/assisted-service:b793c52",
        "discovery-agent": "registry-proxy.engineering.redhat.com/rh-osbs/openshift4-assisted-installer-agent-rhel8:v4.6.0-15",
        "image-builder": "quay.io/app-sre/assisted-iso-create:b793c52"
    }
}

Comment 5 nshidlin 2020-09-18 12:25:10 UTC
Verified on staging 9/18:
{
    "release_tag": "v1.0.9.4-ds",
    "versions": {
        "assisted-ignition-generator": "quay.io/ocpmetal/assisted-ignition-generator:v1.0.9.4",
        "assisted-installer": "registry-proxy.engineering.redhat.com/rh-osbs/openshift4-assisted-installer-rhel8:v4.6.0-19",
        "assisted-installer-controller": "registry-proxy.engineering.redhat.com/rh-osbs/openshift4-assisted-installer-reporter-rhel8:v4.6.0-15",
        "assisted-installer-service": "quay.io/app-sre/assisted-service:b793c52",
        "discovery-agent": "registry-proxy.engineering.redhat.com/rh-osbs/openshift4-assisted-installer-agent-rhel8:v4.6.0-15",
        "image-builder": "quay.io/app-sre/assisted-iso-create:b793c52"
    }
}


Note You need to log in before you can comment on or make changes to this bug.