Bug 1877866 - [Assisted-4.6 ] Cluster installation fails with "*" wildcard in "No Proxy" field
Summary: [Assisted-4.6 ] Cluster installation fails with "*" wildcard in "No Proxy" field
Keywords:
Status: CLOSED DEFERRED
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: assisted-installer
Version: 4.6
Hardware: Unspecified
OS: Unspecified
high
low
Target Milestone: ---
: 4.8.0
Assignee: vemporop
QA Contact: Yuri Obshansky
URL:
Whiteboard: AI-Team-Core
Depends On: 1877486 1947066
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-09-10 16:02 UTC by nshidlin
Modified: 2021-05-26 08:51 UTC (History)
5 users (show)

Fixed In Version: OCP-Metal-V1.0.12.1
Doc Type: Known Issue
Doc Text:
Cause: Several components do not accept an asterisk (*) as a valid no-proxy value. Consequence: A user cannot use '*' to bypass proxy for all destinations. Workaround (if any): Do not use a proxy at all, or specify in no-proxy all the IP addresses, hosts or domains to be excluded. Result: Trying to use '*' results in an error message in the assisted installer.
Clone Of:
Environment:
Last Closed: 2021-05-26 08:51:54 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
screen shot of validation error (103.00 KB, image/png)
2020-09-10 16:02 UTC, nshidlin
no flags Details
cluster failed to start installation in v1.0.15.2 logs (237.50 KB, application/x-tar)
2021-01-28 07:51 UTC, nshidlin
no flags Details
openshift-machine-config-operator.log (6.78 KB, text/plain)
2021-03-31 14:31 UTC, vemporop
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github openshift assisted-service pull 642 0 None closed OCPBUGSM-17327: allow "*" as a valid NoProxyFormat 2021-02-14 23:30:28 UTC
Red Hat Bugzilla 1877486 0 medium CLOSED NoProxy in install-config.yaml doesn't accept '*' 2021-04-08 12:03:12 UTC

Description nshidlin 2020-09-10 16:02:31 UTC
Created attachment 1714451 [details]
screen shot of validation error

Description of problem:
The helper text for the field:
"A comma-separated list of destination domain names, domains, IP addresses or other network CIDRs to exclude proxying. Preface a domain with . to include all subdomains of that domain. Use * to bypass proxy for all destinations."

Entering this value in the UI yeilds the following error:
"NO Proxy format is not valid: '*'. NO Proxy is a comma-separated list of destination domain names, domains, IP addresses or other network CIDRs. A domain can be prefaced with '.' to include all subdomains of that domain."

Attempting to send this field through an API PATCH yeilds the following response:
{
    "code": "400",
    "href": "",
    "id": 400,
    "kind": "Error",
    "reason": "NO Proxy format is not valid: '*'. NO Proxy is a comma-separated list of destination domain names, domains, IP addresses or other network CIDRs. A domain can be prefaced with '.' to include all subdomains of that domain."
}


Version-Release number of selected component (if applicable):
assisted-installer component versions:
{
    "versions": {
        "assisted-ignition-generator": "quay.io/ocpmetal/assisted-ignition-generator:v1.0.9.1",
        "assisted-installer": "quay.io/ocpmetal/assisted-installer:v1.0.9.1",
        "assisted-installer-controller": "quay.io/ocpmetal/assisted-installer-controller:v1.0.9.1",
        "assisted-installer-service": "quay.io/app-sre/assisted-service:f0172a8",
        "discovery-agent": "quay.io/ocpmetal/assisted-installer-agent:v1.0.9.1",
        "image-builder": "quay.io/app-sre/assisted-iso-create:f0172a8"
    }
}

How reproducible:
Every time

Steps to Reproduce:
1. Create a cluster
2. Click "Download Discovery ISO" and input HTTP Proxy URL and set "No Proxy" to "*"
3. Click "Generate HTTP Proxy"

Actual results:
"*" value is rejected

Expected results:
"*" to be a valid option

Additional info:

Comment 1 Moti Asayag 2020-09-13 14:41:51 UTC
This bug is blocked by https://bugzilla.redhat.com/show_bug.cgi?id=1877486 - since 'install-config.yaml' doesn't support '*' as a valid value to bypass proxy, the assisted-service cannot use it as well.

Comment 2 Michael Filanov 2020-09-14 06:48:37 UTC
ealster this ticket should be assigned to slavie

Comment 3 Ronnie Lazar 2020-10-15 11:11:58 UTC
This issue has been fixed in OCP so now we can support this on the Assisted-Service.

Comment 5 nshidlin 2020-12-03 16:13:06 UTC
Failed Verification on staging:
{
    "release_tag": "v1.0.12_1",
    "versions": {
        "assisted-ignition-generator": "",
        "assisted-installer": "registry-proxy.engineering.redhat.com/rh-osbs/openshift4-assisted-installer-rhel8:v4.6.0-49",
        "assisted-installer-controller": "registry-proxy.engineering.redhat.com/rh-osbs/openshift4-assisted-installer-reporter-rhel8:v4.6.0-46",
        "assisted-installer-service": "quay.io/app-sre/assisted-service:92e3dad",
        "discovery-agent": "registry-proxy.engineering.redhat.com/rh-osbs/openshift4-assisted-installer-agent-rhel8:v4.6.0-47",
        "image-builder": "quay.io/app-sre/assisted-iso-create:92e3dad"
    }
}

Steps to reproduce:
1. Set http proxy url to a non-existent proxy server
2. Set no proxy value to bypass all destinations "*"
3. Start cluster install

Result:
Cluster install fails with:
2/3 masters(non-bootstrap) installed
2/2 worker nodes failed on "This host failed its installation.
Host failed to install because its installation stage Waiting for ignition took longer than expected 1h0m0s"

Comment 6 nshidlin 2021-01-28 07:51:25 UTC
Created attachment 1751603 [details]
cluster failed to start installation in v1.0.15.2 logs

Comment 7 nshidlin 2021-01-28 07:53:09 UTC
Updated behavior: 
{
    "release_tag": "v1.0.15.2",
    "versions": {
        "assisted-installer": "registry-proxy.engineering.redhat.com/rh-osbs/openshift4-assisted-installer-rhel8:v4.6.0-82",
        "assisted-installer-controller": "registry-proxy.engineering.redhat.com/rh-osbs/openshift4-assisted-installer-reporter-rhel8:v4.6.0-81",
        "assisted-installer-service": "quay.io/app-sre/assisted-service:3b47299",
        "discovery-agent": "registry-proxy.engineering.redhat.com/rh-osbs/openshift4-assisted-installer-agent-rhel8:v4.6.0-86"
    }
}

Steps to reproduce:
1. Set http proxy url to a non-existent proxy server
2. Set no proxy value to bypass all destinations "*"
3. Start cluster install

Result:
Installation fails to start:
Failed generating kubeconfig files for cluster cf83e415-dbf3-4af1-9867-97c47f4603a4: exit status 1.
Reset the installation process to return to the configuration and try again. Some hosts may need to be re-registered by rebooting into the Discovery ISO.

logs are attached

Comment 8 Angus Salkeld 2021-02-14 23:33:19 UTC
The PR below improves the situation greatly, but the cluster is left with
1. the master that was the bootstrap doesn't come up after it reboots
2. the workers don't come up correctly
 
https://github.com/openshift/assisted-service/pull/1035

Comment 9 Angus Salkeld 2021-03-22 21:35:51 UTC
I am working on metal3 stuff now, so unlikely to have time to work further on this.

Comment 10 vemporop 2021-03-30 09:26:59 UTC
An asterisk '*' causes the openshift-install command to fail as below. This is because we will automatically append CIDRs and VIPs to a noProxy value, which results in something like '*,192.168.126.0/24,.test-infra-cluster-assisted-installer.redhat.com,10.128.0.0/14,172.30.0.0/16'. An asterisk in noProxy alone works fine.

Comment 11 vemporop 2021-03-31 14:31:01 UTC
Created attachment 1768103 [details]
openshift-machine-config-operator.log

After I fixed the assisted service side, two masters of a cluster are installed successfully but the bootstrap fails to download ignition from the API VIP, because machine-config-operator crashes. Log attached.

Comment 12 vemporop 2021-04-07 15:41:34 UTC
Blocked by https://bugzilla.redhat.com/show_bug.cgi?id=1947066 (machine-config-operator pod crashes when noProxy is *)

Comment 13 vemporop 2021-05-20 13:01:39 UTC
Until a fix for the blocking bug https://bugzilla.redhat.com/show_bug.cgi?id=1947066 is released, I've disabled '*' as a valid noProxy value in the assisted installer. So now a user gets a clear error message when trying to enter it, instead of the confusing installer failure.

Comment 14 vemporop 2021-05-25 05:47:32 UTC
A fix for https://bugzilla.redhat.com/show_bug.cgi?id=1947066 is part of OCP 4.8.0-fc.4, but both fc.4 and fc.5 have been rejected because of issues. So we'll have to wait for the first good OCP version after fc.4.

Comment 15 Ronnie Lazar 2021-05-26 08:51:54 UTC
For now we have disabled this option.
Will handle re-adding this in epic https://issues.redhat.com/browse/MGMT-6647


Note You need to log in before you can comment on or make changes to this bug.