Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1862223

Summary: [Assisted-4.5-M2] Node discovering failed since agent starts with wrong host parameter on PSI
Product: OpenShift Container Platform Reporter: Yuri Obshansky <yobshans>
Component: assisted-installerAssignee: Yuri Obshansky <yobshans>
assisted-installer sub component: discovery-agent QA Contact: Yuri Obshansky <yobshans>
Status: CLOSED NOTABUG Docs Contact:
Severity: urgent    
Priority: urgent CC: alazar, aos-bugs, rfreiman, sasha
Version: 4.5   
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-08-03 20:05:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Yuri Obshansky 2020-07-30 19:25:13 UTC 
Description of problem:
QE has separate Assisted Installer running on PSI
http://assisted-installer-ui.apps.ocp.prod.psi.redhat.com/clusters
Node discovering failed because agent starts with wrong host. It points to Dev Assisted Installer Beta on PSI.
[core@master-0-0 log]$ systemctl status agent
● agent.service
   Loaded: loaded (/etc/systemd/system/agent.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2020-07-30 19:08:39 UTC; 12min ago
  Process: 1484 ExecStartPre=/usr/bin/podman run --privileged --rm -v /usr/local/bin:/hostbin quay.io/ocpmetal/agent:latest cp /usr/bin/agent /ho>
 Main PID: 1961 (agent)
    Tasks: 13 (limit: 204943)
   Memory: 311.9M
   CGroup: /system.slice/agent.service
           └─1961 /usr/local/bin/agent --host assisted-installer.apps.ocp.prod.psi.redhat.com --port 80 --cluster-id e860dc6f-3b8c-47d4-aa0b-3711>

It should point to assisted-installer-ui.apps.ocp.prod.psi.redhat.com/clusters instead of assisted-installer.apps.ocp.prod.psi.redhat.com

As result agent throws error:
Error registering host: [POST /clusters/{cluster_id}/hosts][400] registerHostBadRequest  &{Code:0xc000504150 Href:0xc000504160 ID:0xc00039628c Kind:0xc000504170 Reason:0xc000504180}" file="register_node.go:43" request_id=d6273afa-44de-4ed0-b435-f40110b2edd0

I verified our ansible playbook and it works correct.
Here is output:
TASK [Creates a new OpenShift bare metal cluster definition.] ************************************************************************************
task path: /home/slave1/jenkins/workspace/ocp-assisted/ocp-edge-qe-demo/linchpin-workspace/hooks/ansible/ocp-edge-setup/assisted-deploy-ocp-cluster.yaml:19
ok: [localhost] => {"changed": false, "connection": "close", "content": "{\"base_dns_domain\":\"qe.lab.redhat.com\",\"cluster_network_cidr\":\"10.128.0.0/14\",\"cluster_network_host_prefix\":23,\"created_at\":\"2020-07-30T19:04:11\",\"host_networks\":null,\"hosts\":null,\"href\":\"/api/assisted-install/v1/clusters/e860dc6f-3b8c-47d4-aa0b-3711ecc3923b\",\"id\":\"e860dc6f-3b8c-47d4-aa0b-3711ecc3923b\",\"image_info\":{\"created_at\":\"0001-01-01T00:00:00\"},\"install_completed_at\":\"2000-01-01T00:00:00\",\"install_started_at\":\"2000-01-01T00:00:00\",\"kind\":\"Cluster\",\"name\":\"seal15-ocp-cluster-0\",\"openshift_version\":\"4.5\",\"org_id\":\"0000000\",\"pull_secret_set\":true,\"service_network_cidr\":\"172.30.0.0/16\",\"ssh_public_key\":\"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDB/BByoBTYNQj/FtBxAQQnyU5DZOEIuKlUTtR1UElmb8iOOYs67PutK1xYSa+5kCUIdVRChsvq4jNqyoSG3S88ELO1shIuYQSaOM8vAGnH7dQSpXHlJUfux7utog+6oRbMZxApGctqvzn/w7GEMSQB4zoCQBXwTn+B8jXX5UFXffb6yOmjKcflFgG5/W+nUfDKIhmp7CzN6G0qCRdug2brq0dVwdlK5qYS1Rr18Hg9+WYglwcUv1ET2Be1DZCbvBSxPPoHh5QWKss0urNmxiWZdlzazuSgxMSs/Rnvd8uIpogxE9lKCfEdvdbhgem6d5XvKL7JDAq3IDA+cS8/Afu6HABBQt/WsKMUCBaWWA96CROaEprpBxlV/Q8lLRt3DkgycbVGTIq+JVGMuUMOW8USNJ0FD23kGJSRC+cbmjDs6co5AHDFXyPJocIYei6mTMJ+wUM6K0zRqgG8jcZkobK84xNYrTY0h8p4I2d9duPRW7QvdJjq/4MttcPZSCglsCM= root.lab.tlv.redhat.com\",\"status\":\"insufficient\",\"status_info\":\"cluster is insufficient, exactly 3 known master hosts are needed for installation\",\"status_updated_at\":\"2020-07-30T19:04:11\",\"updated_at\":\"2020-07-30T19:04:11\",\"user_id\":\"0000000\"}\n", "content_length": "1422", "content_type": "application/json", "cookies": {"7c441cd171f2243415fc7df491f023c8": "7a6187f5a3bb6ef86c864edb48607164"}, "cookies_string": "7c441cd171f2243415fc7df491f023c8=7a6187f5a3bb6ef86c864edb48607164", "date": "Thu, 30 Jul 2020 19:04:12 GMT", "elapsed": 0, "json": {"base_dns_domain": "qe.lab.redhat.com", "cluster_network_cidr": "10.128.0.0/14", "cluster_network_host_prefix": 23, "created_at": "2020-07-30T19:04:11", "host_networks": null, "hosts": null, "href": "/api/assisted-install/v1/clusters/e860dc6f-3b8c-47d4-aa0b-3711ecc3923b", "id": "e860dc6f-3b8c-47d4-aa0b-3711ecc3923b", "image_info": {"created_at": "0001-01-01T00:00:00"}, "install_completed_at": "2000-01-01T00:00:00", "install_started_at": "2000-01-01T00:00:00", "kind": "Cluster", "name": "seal15-ocp-cluster-0", "openshift_version": "4.5", "org_id": "0000000", "pull_secret_set": true, "service_network_cidr": "172.30.0.0/16", "ssh_public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDB/BByoBTYNQj/FtBxAQQnyU5DZOEIuKlUTtR1UElmb8iOOYs67PutK1xYSa+5kCUIdVRChsvq4jNqyoSG3S88ELO1shIuYQSaOM8vAGnH7dQSpXHlJUfux7utog+6oRbMZxApGctqvzn/w7GEMSQB4zoCQBXwTn+B8jXX5UFXffb6yOmjKcflFgG5/W+nUfDKIhmp7CzN6G0qCRdug2brq0dVwdlK5qYS1Rr18Hg9+WYglwcUv1ET2Be1DZCbvBSxPPoHh5QWKss0urNmxiWZdlzazuSgxMSs/Rnvd8uIpogxE9lKCfEdvdbhgem6d5XvKL7JDAq3IDA+cS8/Afu6HABBQt/WsKMUCBaWWA96CROaEprpBxlV/Q8lLRt3DkgycbVGTIq+JVGMuUMOW8USNJ0FD23kGJSRC+cbmjDs6co5AHDFXyPJocIYei6mTMJ+wUM6K0zRqgG8jcZkobK84xNYrTY0h8p4I2d9duPRW7QvdJjq/4MttcPZSCglsCM= root.lab.tlv.redhat.com", "status": "insufficient", "status_info": "cluster is insufficient, exactly 3 known master hosts are needed for installation", "status_updated_at": "2020-07-30T19:04:11", "updated_at": "2020-07-30T19:04:11", "user_id": "0000000"}, "msg": "OK (1422 bytes)", "redirected": false, "server": "nginx", "set_cookie": "7c441cd171f2243415fc7df491f023c8=7a6187f5a3bb6ef86c864edb48607164; path=/; HttpOnly", "status": 201, "url": "http://assisted-installer-ui.apps.ocp.prod.psi.redhat.com/api/assisted-install/v1/clusters", "x_frame_options": "SAMEORIGIN"}

TASK [Creates a new OpenShift per-cluster discovery ISO.] ****************************************************************************************
task path: /home/slave1/jenkins/workspace/ocp-assisted/ocp-edge-qe-demo/linchpin-workspace/hooks/ansible/ocp-edge-setup/assisted-deploy-ocp-cluster.yaml:41
ok: [localhost] => {"changed": false, "connection": "close", "content": "{\"base_dns_domain\":\"qe.lab.redhat.com\",\"cluster_network_cidr\":\"10.128.0.0/14\",\"cluster_network_host_prefix\":23,\"created_at\":\"2020-07-30T19:04:11\",\"host_networks\":null,\"hosts\":[],\"href\":\"/api/assisted-install/v1/clusters/e860dc6f-3b8c-47d4-aa0b-3711ecc3923b\",\"id\":\"e860dc6f-3b8c-47d4-aa0b-3711ecc3923b\",\"image_info\":{\"created_at\":\"2020-07-30T19:04:13\",\"generator_version\":\"quay.io/ocpmetal/installer-image-build:latest\",\"ssh_public_key\":\"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDB/BByoBTYNQj/FtBxAQQnyU5DZOEIuKlUTtR1UElmb8iOOYs67PutK1xYSa+5kCUIdVRChsvq4jNqyoSG3S88ELO1shIuYQSaOM8vAGnH7dQSpXHlJUfux7utog+6oRbMZxApGctqvzn/w7GEMSQB4zoCQBXwTn+B8jXX5UFXffb6yOmjKcflFgG5/W+nUfDKIhmp7CzN6G0qCRdug2brq0dVwdlK5qYS1Rr18Hg9+WYglwcUv1ET2Be1DZCbvBSxPPoHh5QWKss0urNmxiWZdlzazuSgxMSs/Rnvd8uIpogxE9lKCfEdvdbhgem6d5XvKL7JDAq3IDA+cS8/Afu6HABBQt/WsKMUCBaWWA96CROaEprpBxlV/Q8lLRt3DkgycbVGTIq+JVGMuUMOW8USNJ0FD23kGJSRC+cbmjDs6co5AHDFXyPJocIYei6mTMJ+wUM6K0zRqgG8jcZkobK84xNYrTY0h8p4I2d9duPRW7QvdJjq/4MttcPZSCglsCM= root.lab.tlv.redhat.com\"},\"install_completed_at\":\"2000-01-01T00:00:00\",\"install_started_at\":\"2000-01-01T00:00:00\",\"kind\":\"Cluster\",\"name\":\"seal15-ocp-cluster-0\",\"openshift_version\":\"4.5\",\"org_id\":\"0000000\",\"pull_secret_set\":true,\"service_network_cidr\":\"172.30.0.0/16\",\"ssh_public_key\":\"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDB/BByoBTYNQj/FtBxAQQnyU5DZOEIuKlUTtR1UElmb8iOOYs67PutK1xYSa+5kCUIdVRChsvq4jNqyoSG3S88ELO1shIuYQSaOM8vAGnH7dQSpXHlJUfux7utog+6oRbMZxApGctqvzn/w7GEMSQB4zoCQBXwTn+B8jXX5UFXffb6yOmjKcflFgG5/W+nUfDKIhmp7CzN6G0qCRdug2brq0dVwdlK5qYS1Rr18Hg9+WYglwcUv1ET2Be1DZCbvBSxPPoHh5QWKss0urNmxiWZdlzazuSgxMSs/Rnvd8uIpogxE9lKCfEdvdbhgem6d5XvKL7JDAq3IDA+cS8/Afu6HABBQt/WsKMUCBaWWA96CROaEprpBxlV/Q8lLRt3DkgycbVGTIq+JVGMuUMOW8USNJ0FD23kGJSRC+cbmjDs6co5AHDFXyPJocIYei6mTMJ+wUM6K0zRqgG8jcZkobK84xNYrTY0h8p4I2d9duPRW7QvdJjq/4MttcPZSCglsCM= root.lab.tlv.redhat.com\",\"status\":\"insufficient\",\"status_info\":\"cluster is insufficient, exactly 3 known master hosts are needed for installation\",\"status_updated_at\":\"2020-07-30T19:04:11\",\"updated_at\":\"2020-07-30T19:04:13\",\"user_id\":\"0000000\"}\n", "content_type": "application/json", "cookies": {"7c441cd171f2243415fc7df491f023c8": "7a6187f5a3bb6ef86c864edb48607164"}, "cookies_string": "7c441cd171f2243415fc7df491f023c8=7a6187f5a3bb6ef86c864edb48607164", "date": "Thu, 30 Jul 2020 19:05:06 GMT", "elapsed": 53, "json": {"base_dns_domain": "qe.lab.redhat.com", "cluster_network_cidr": "10.128.0.0/14", "cluster_network_host_prefix": 23, "created_at": "2020-07-30T19:04:11", "host_networks": null, "hosts": [], "href": "/api/assisted-install/v1/clusters/e860dc6f-3b8c-47d4-aa0b-3711ecc3923b", "id": "e860dc6f-3b8c-47d4-aa0b-3711ecc3923b", "image_info": {"created_at": "2020-07-30T19:04:13", "generator_version": "quay.io/ocpmetal/installer-image-build:latest", "ssh_public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDB/BByoBTYNQj/FtBxAQQnyU5DZOEIuKlUTtR1UElmb8iOOYs67PutK1xYSa+5kCUIdVRChsvq4jNqyoSG3S88ELO1shIuYQSaOM8vAGnH7dQSpXHlJUfux7utog+6oRbMZxApGctqvzn/w7GEMSQB4zoCQBXwTn+B8jXX5UFXffb6yOmjKcflFgG5/W+nUfDKIhmp7CzN6G0qCRdug2brq0dVwdlK5qYS1Rr18Hg9+WYglwcUv1ET2Be1DZCbvBSxPPoHh5QWKss0urNmxiWZdlzazuSgxMSs/Rnvd8uIpogxE9lKCfEdvdbhgem6d5XvKL7JDAq3IDA+cS8/Afu6HABBQt/WsKMUCBaWWA96CROaEprpBxlV/Q8lLRt3DkgycbVGTIq+JVGMuUMOW8USNJ0FD23kGJSRC+cbmjDs6co5AHDFXyPJocIYei6mTMJ+wUM6K0zRqgG8jcZkobK84xNYrTY0h8p4I2d9duPRW7QvdJjq/4MttcPZSCglsCM= root.lab.tlv.redhat.com"}, "install_completed_at": "2000-01-01T00:00:00", "install_started_at": "2000-01-01T00:00:00", "kind": "Cluster", "name": "seal15-ocp-cluster-0", "openshift_version": "4.5", "org_id": "0000000", "pull_secret_set": true, "service_network_cidr": "172.30.0.0/16", "ssh_public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDB/BByoBTYNQj/FtBxAQQnyU5DZOEIuKlUTtR1UElmb8iOOYs67PutK1xYSa+5kCUIdVRChsvq4jNqyoSG3S88ELO1shIuYQSaOM8vAGnH7dQSpXHlJUfux7utog+6oRbMZxApGctqvzn/w7GEMSQB4zoCQBXwTn+B8jXX5UFXffb6yOmjKcflFgG5/W+nUfDKIhmp7CzN6G0qCRdug2brq0dVwdlK5qYS1Rr18Hg9+WYglwcUv1ET2Be1DZCbvBSxPPoHh5QWKss0urNmxiWZdlzazuSgxMSs/Rnvd8uIpogxE9lKCfEdvdbhgem6d5XvKL7JDAq3IDA+cS8/Afu6HABBQt/WsKMUCBaWWA96CROaEprpBxlV/Q8lLRt3DkgycbVGTIq+JVGMuUMOW8USNJ0FD23kGJSRC+cbmjDs6co5AHDFXyPJocIYei6mTMJ+wUM6K0zRqgG8jcZkobK84xNYrTY0h8p4I2d9duPRW7QvdJjq/4MttcPZSCglsCM= root.lab.tlv.redhat.com", "status": "insufficient", "status_info": "cluster is insufficient, exactly 3 known master hosts are needed for installation", "status_updated_at": "2020-07-30T19:04:11", "updated_at": "2020-07-30T19:04:13", "user_id": "0000000"}, "msg": "OK (unknown bytes)", "redirected": false, "server": "nginx", "set_cookie": "7c441cd171f2243415fc7df491f023c8=7a6187f5a3bb6ef86c864edb48607164; path=/; HttpOnly", "status": 201, "transfer_encoding": "chunked", "url": "http://assisted-installer-ui.apps.ocp.prod.psi.redhat.com/api/assisted-install/v1/clusters/e860dc6f-3b8c-47d4-aa0b-3711ecc3923b/downloads/image", "x_frame_options": "SAMEORIGIN"}


Version-Release number of selected component (if applicable):
Assisted Installer UI version
    refs/heads/master (quay.io/ocpmetal/ocp-metal-ui:6ea08686e6236151f6d9f91fc78c16e431e10f4c)
Assisted Installer
    quay.io/ocpmetal/assisted-installer:latest
Assisted Installer Controller
    quay.io/ocpmetal/assisted-installer-controller:latest
Assisted Installer Service
    quay.io/ocpmetal/bm-inventory:latest
Discovery Agent
    quay.io/ocpmetal/agent:latest
Ignition Manifests and Kubeconfig Generate
    quay.io/ocpmetal/ignition-manifests-and-kubeconfig-generate:latest
Image Builder
    quay.io/ocpmetal/installer-image-build:latest

How reproducible:
http://assisted-installer-ui.apps.ocp.prod.psi.redhat.com/clusters/e860dc6f-3b8c-47d4-aa0b-3711ecc3923b

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 1 Yuri Obshansky 2020-07-31 18:06:23 UTC 
So, finally this is an environmental issue and has been resolved by manual editing the bm-inventory configmap.
Probably we need an additional parameter for command: make deploy-all which will allow set INVENTORY_URL.

Here is config map yaml before fix

kind: ConfigMap
apiVersion: v1
metadata:
  name: bm-inventory-config
  namespace: assisted-installer-qe
  selfLink: /api/v1/namespaces/assisted-installer-qe/configmaps/bm-inventory-config
  uid: fa61eb74-3885-4ddc-a32e-d13c9e711715
  resourceVersion: '121716639'
  creationTimestamp: '2020-07-23T18:57:42Z'
  labels:
    app: bm-inventory
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"v1","data":{"AGENT_DOCKER_IMAGE":"quay.io/ocpmetal/agent:latest","BASE_DNS_DOMAINS":"","CONNECTIVITY_CHECK_IMAGE":"quay.io/ocpmetal/connectivity_check:latest","CONTROLLER_IMAGE":"quay.io/ocpmetal/assisted-installer-controller:latest","IMAGE_BUILDER":"quay.io/ocpmetal/installer-image-build:latest","INSTALLER_IMAGE":"quay.io/ocpmetal/assisted-installer:latest","INVENTORY_IMAGE":"quay.io/ocpmetal/inventory:latest","INVENTORY_PORT":"80","INVENTORY_URL":"assisted-installer.apps.ocp.prod.psi.redhat.com","KUBECONFIG_GENERATE_IMAGE":"quay.io/ocpmetal/ignition-manifests-and-kubeconfig-generate:latest","NAMESPACE":"assisted-installer-qe","OPENSHIFT_INSTALL_RELEASE_IMAGE":"quay.io/openshift-release-dev/ocp-release@sha256:eab93b4591699a5a4ff50ad3517892653f04fb840127895bb3609b3cc68f98f3","SELF_VERSION":"quay.io/ocpmetal/bm-inventory:latest"},"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"app":"bm-inventory"},"name":"bm-inventory-config","namespace":"assisted-installer-qe"}}
data:
  KUBECONFIG_GENERATE_IMAGE: 'quay.io/ocpmetal/ignition-manifests-and-kubeconfig-generate:latest'
  INVENTORY_PORT: '80'
  INSTALLER_IMAGE: 'quay.io/ocpmetal/assisted-installer:latest'
  SELF_VERSION: 'quay.io/ocpmetal/bm-inventory:latest'
  BASE_DNS_DOMAINS: ''
  INVENTORY_URL: assisted-installer.apps.ocp.prod.psi.redhat.com
  AGENT_DOCKER_IMAGE: 'quay.io/ocpmetal/agent:latest'
  IMAGE_BUILDER: 'quay.io/ocpmetal/installer-image-build:latest'
  INVENTORY_IMAGE: 'quay.io/ocpmetal/inventory:latest'
  OPENSHIFT_INSTALL_RELEASE_IMAGE: >-
    quay.io/openshift-release-dev/ocp-release@sha256:eab93b4591699a5a4ff50ad3517892653f04fb840127895bb3609b3cc68f98f3
  CONTROLLER_IMAGE: 'quay.io/ocpmetal/assisted-installer-controller:latest'
  NAMESPACE: assisted-installer-qe
  CONNECTIVITY_CHECK_IMAGE: 'quay.io/ocpmetal/connectivity_check:latest'
Comment 2 Rom Freiman 2020-08-03 17:54:49 UTC 
It's not a bug. Please close