The spoke cluster in this bug is SNO.

@sasha can you please attach logs from the service? 
When uploading there should be a log like: https://github.com/openshift/assisted-service/blob/master/internal/bminventory/inventory.go#L4213

At some points we don’t have the logs yet but the URL is still there, for example create a new cluster and not start the installation yet. Adding a condition that will indicate the state of the logs will help use improve the UX

for example in the cluster we have a collection date of the logs
https://github.com/openshift/assisted-service/blob/master/swagger.yaml#L4248-L4255

The link started the work only when the cluster state switched to:
"The installation is in progress: Finalizing cluster installation"


curl -k https://assisted-service-rhacm.apps.sasha-0.qe.lab.redhat.com/api/assisted-install/v1/clusters/6d2ee156-ab56-4b63-a12d-bbf61854bdbc/logs?api_key=eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbHVzdGVyX2lkIjoiNmQyZWUxNTYtYWI1Ni00YjYzLWExMmQtYmJmNjE4NTRiZGJjIn0.yd6inq87lEe-YNuk6bIXW9SfjJYs0mEaZjB0tW1rnjPL-PzEkpr4HchUS9bg2JJPLxdTZ36FwmEmz2K1zEGWZg
Warning: Binary output can mess up your terminal. Use "--output -" to tell 
Warning: curl to output it to your terminal anyway, or consider "--output 
Warning: <FILE>" to save to a file.

The way to communicate between backend and kube-api is currently done with events, at the moment we don't have events that indicate logs upload and we don't have support for debug events (don't want to spam the users with log events) 
so for the moment this task will move to 4.9

switch to medium , let me know if need to bump it back to urgent.

https://github.com/openshift/assisted-service/pull/2241 merged to master.

QE can't currently verify this - no version for them to test against.

Moving to assigned in hope that the bot won't change to on_qa.

Should be kept as ON_QA since the fix was merged.
It is a 4.9 target release bug, so it is not expected that QE will verify this before they start working on 4.9.

Version:
4.9.0-0.nightly-2021-08-31-123131
quay.io/acm-d/acm-custom-registry:2.4.0-DOWNSTREAM-2021-08-31-23-32-56


Deployed spoke cluster version quay.io/openshift-release-dev/ocp-release:4.9.0-fc.1-x86_64


The issue still reproduces:

oc describe agentclusterinstalls.extensions.hive.openshift.io 
Name:         elvis2
Namespace:    elvis2
Labels:       <none>
Annotations:  <none>
API Version:  extensions.hive.openshift.io/v1beta1
Kind:         AgentClusterInstall
Metadata:
  Creation Timestamp:  2021-09-03T13:42:37Z
  Finalizers:
    agentclusterinstall.agent-install.openshift.io/ai-deprovision
  Generation:  2
  Managed Fields:
    API Version:  extensions.hive.openshift.io/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        .:
        f:clusterDeploymentRef:
          .:
          f:name:
        f:imageSetRef:
          .:
          f:name:
        f:networking:
          .:
          f:clusterNetwork:
          f:machineNetwork:
          f:serviceNetwork:
        f:provisionRequirements:
          .:
          f:controlPlaneAgents:
        f:sshPublicKey:
    Manager:      kubectl-create
    Operation:    Update
    Time:         2021-09-03T13:42:37Z
    API Version:  extensions.hive.openshift.io/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        f:conditions:
        f:debugInfo:
          f:state:
          f:stateInfo:
    Manager:    assisted-service
    Operation:  Update
    Time:       2021-09-03T13:53:06Z
  Owner References:
    API Version:     hive.openshift.io/v1
    Kind:            ClusterDeployment
    Name:            elvis2
    UID:             4b138f9e-a04a-46eb-b568-9f1e15e5e8c2
  Resource Version:  2302132
  UID:               b730d4b2-66b6-4ecf-9a4d-093a912f47c8
Spec:
  Cluster Deployment Ref:
    Name:  elvis2
  Cluster Metadata:
    Admin Kubeconfig Secret Ref:
      Name:  elvis2-admin-kubeconfig
    Admin Password Secret Ref:
    Cluster ID:  ec24779b-67ca-4be9-9598-9a4ded3d0dfe
    Infra ID:    281b3d8e-39f5-4c05-ab3c-fe107ed28de4
  Image Set Ref:
    Name:  4.8
  Networking:
    Cluster Network:
      Cidr:         10.128.0.0/14
      Host Prefix:  23
    Machine Network:
      Cidr:  192.168.123.0/24
    Service Network:
      172.30.0.0/16
  Provision Requirements:
    Control Plane Agents:  1
  Ssh Public Key:          ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCzwAz3fnZcrca7mY/kVFpQGS2yI1uGd/+t3PMJn/C7Ppj1uIG32ufHkTq+SXh8Zg3xcy9v/Uome1mo3FP7PoGsWms5B9wzbooGhbA3rdph0/NxSzrHO3qcudcJsBM4GVJhcbFfbkzJVCPZQ94O/Y17oKjKuaBz69clPD29BlzKF4xCWzzbJW5Q8Y9tvWvDpCdVBM7VorpAn3MaA95xL6e15douWwwlhdI4dIOk/+8HcfgJnZGyOeLTnLVpjxQaFzTj3ScEud/5yd5wHcICrHH8Fbq419nN7VWjxbMNWUn182mcCCs0RXx2eyYq27yJvgkJS86n09SyLynX6ySqkFXN
Status:
  Conditions:
    Last Probe Time:             2021-09-03T13:42:45Z
    Last Transition Time:        2021-09-03T13:42:45Z
    Message:                     SyncOK
    Reason:                      SyncOK
    Status:                      True
    Type:                        SpecSynced
    Last Probe Time:             2021-09-03T13:51:14Z
    Last Transition Time:        2021-09-03T13:51:14Z
    Message:                     The cluster's validations are passing
    Reason:                      ValidationsPassing
    Status:                      True
    Type:                        Validated
    Last Probe Time:             2021-09-03T13:51:14Z
    Last Transition Time:        2021-09-03T13:51:14Z
    Message:                     The cluster requirements are met
    Reason:                      ClusterAlreadyInstalling
    Status:                      True
    Type:                        RequirementsMet
    Last Probe Time:             2021-09-03T13:53:06Z
    Last Transition Time:        2021-09-03T13:53:06Z
    Message:                     The installation is in progress: Installation in progress
    Reason:                      InstallationInProgress
    Status:                      False
    Type:                        Completed
    Last Probe Time:             2021-09-03T13:42:45Z
    Last Transition Time:        2021-09-03T13:42:45Z
    Message:                     The installation has not failed
    Reason:                      InstallationNotFailed
    Status:                      False
    Type:                        Failed
    Last Probe Time:             2021-09-03T13:42:45Z
    Last Transition Time:        2021-09-03T13:42:45Z
    Message:                     The installation is waiting to start or in progress
    Reason:                      InstallationNotStopped
    Status:                      False
    Type:                        Stopped
  Connectivity Majority Groups:  {"192.168.123.0/24":[]}
  Debug Info:
    Events URL:  https://assisted-service-rhacm.apps.sasha-0.qe.lab.redhat.com/api/assisted-install/v1/clusters/281b3d8e-39f5-4c05-ab3c-fe107ed28de4/events?api_key=eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbHVzdGVyX2lkIjoiMjgxYjNkOGUtMzlmNS00YzA1LWFiM2MtZmUxMDdlZDI4ZGU0In0.9rvq5EHKjGN82z1JfSbHihGyZhKRo86G10_Wh5kSKy1jTu4tHxjHdFjCwvqOaWZCXJkhxvp_gv3qYepiDX4yVw
    Logs URL:    https://assisted-service-rhacm.apps.sasha-0.qe.lab.redhat.com/api/assisted-install/v1/clusters/281b3d8e-39f5-4c05-ab3c-fe107ed28de4/logs?api_key=eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbHVzdGVyX2lkIjoiMjgxYjNkOGUtMzlmNS00YzA1LWFiM2MtZmUxMDdlZDI4ZGU0In0.dgsqAdMxUvBKj2RHLqJKptLwj33vhZNbXadP93de3fE5uvrtOrYx_lYQCq-FhD_gb0iBuenDV0Lc9m2lAzE16w
    State:       installing
    State Info:  Installation in progress
Events:          <none>



curl -k https://assisted-service-rhacm.apps.sasha-0.qe.lab.redhat.com/api/assisted-install/v1/clusters/281b3d8e-39f5-4c05-ab3c-fe107ed28de4/logs?api_key=eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbHVzdGVyX2lkIjoiMjgxYjNkOGUtMzlmNS00YzA1LWFiM2MtZmUxMDdlZDI4ZGU0In0.dgsqAdMxUvBKj2RHLqJKptLwj33vhZNbXadP93de3fE5uvrtOrYx_lYQCq-FhD_gb0iBuenDV0Lc9m2lAzE16w
{"code":"500","href":"","id":500,"kind":"Error","reason":"No log files were found"}

Once the cluster being deployed switched to finalizing, the link started working.


curl -k https://assisted-service-rhacm.apps.sasha-0.qe.lab.redhat.com/api/assisted-install/v1/clusters/281b3d8e-39f5-4c05-ab3c-fe107ed28de4/logs?api_key=eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbHVzdGVyX2lkIjoiMjgxYjNkOGUtMzlmNS00YzA1LWFiM2MtZmUxMDdlZDI4ZGU0In0.dgsqAdMxUvBKj2RHLqJKptLwj33vhZNbXadP93de3fE5uvrtOrYx_lYQCq-FhD_gb0iBuenDV0Lc9m2lAzE16w
Warning: Binary output can mess up your terminal. Use "--output -" to tell 
Warning: curl to output it to your terminal anyway, or consider "--output 
Warning: <FILE>" to save to a file.

I have just repeated the test.
I see logs available from much before entering the finalize state:

#  kubectl -n nir-test1 get agentclusterinstall test-agent-cluster-install -o=jsonpath="{.metadata.name}{'\n'}{range .status.conditions[*]}{.type}{'\t'}{'\t'}{.message}{'\n'}"
test-agent-cluster-install
SpecSynced		SyncOK
Validated		The cluster's validations are passing
RequirementsMet		The cluster requirements are met
Completed		The installation is in progress: Installation in progress
Failed		The installation has not failed
Stopped		The installation is waiting to start or in progress


Getting the logs:
=================
# kubectl -n nir-test1 get aci -o yaml | grep logsURL
      logsURL: https://assisted-service-assisted-installer.apps.ostest.test.metalkube.org/api/assisted-install/v1/clusters/bf096e69-d944-448c-b2cc-ea04f9fc15aa/logs?api_key=eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbHVzdGVyX2lkIjoiYmYwOTZlNjktZDk0NC00NDhjLWIyY2MtZWEwNGY5ZmMxNWFhIn0.9D5AFzCYWR1bk3ZvP9FXj8Vs9qrH0PIvAuur4VmxAb3hR_tgNxs_mUj7JxXvK0tPBCjoeDSVEMV5h-pwRrmC0w


# curl -k https://assisted-service-assisted-installer.apps.ostest.test.metalkube.org/api/assisted-install/v1/clusters/bf096e69-d944-448c-b2cc-ea04f9fc15aa/logs?api_key=eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbHVzdGVyX2lkIjoiYmYwOTZlNjktZDk0NC00NDhjLWIyY2MtZWEwNGY5ZmMxNWFhIn0.9D5AFzCYWR1bk3ZvP9FXj8Vs9qrH0PIvAuur4VmxAb3hR_tgNxs_mUj7JxXvK0tPBCjoeDSVEMV5h-pwRrmC0w --output foo.tar

# tar -xvf foo.tar
controller_logs.tar.gz

# tar -xvf controller_logs.tar.gz
assisted-installer-controller-5464x.logs

# wc -l assisted-installer-controller-5464x.logs
64 assisted-installer-controller-5464x.logs


Tested with dev-scripts, deploying a 3 nodes cluster.


Version (master):
=================
# kubectl -n assisted-installer describe pod assisted-service-c47677549-27wxk  | grep -i image
    Image:          quay.io/ocpmetal/assisted-service:latest
    Image ID:       quay.io/ocpmetal/assisted-service@sha256:3e7ef95232d64eb79cb8ae52615d2353827e7f29ae457a5055d6048948232bc7
    Image:          quay.io/ocpmetal/postgresql-12-centos7:latest
    Image ID:       quay.io/ocpmetal/postgresql-12-centos7@sha256:94727d70e0afbf4e167e078744f3a10ac9d82edc553d57b0ecbb5443264f07e1


Can you confirm this happens with the latest code?

As you can see below only when the finalizing reached 85% the link started working:

HUB OCP version:
4.9.0-0.nightly-2021-09-10-170926

ACM vesion:
quay.io/acm-d/acm-custom-registry:2.4.0-DOWNSTREAM-2021-09-13-02-14-41



oc get AgentClusterInstall -n elvis2 elvis2 -o=custom-columns='STATUS:status.conditions[-3].message' 
STATUS
The installation is in progress: Installation in progress

curl -k https://assisted-service-rhacm.apps.sasha-0.qe.lab.redhat.com/api/assisted-install/v1/clusters/5dd0eff4-cf65-4425-a35e-583d5f40a5a5/logs?api_key=eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbHVzdGVyX2lkIjoiNWRkMGVmZjQtY2Y2NS00NDI1LWEzNWUtNTgzZDVmNDBhNWE1In0.uYTH7FbBvv0CSExYihj75UYoTUcFkWOTho5pHTjL0WC4KV86ubdts9ytDE92nOu_429bTpOJIXM81RMoGwHXXA
{"code":"500","href":"","id":500,"kind":"Error","reason":"No log files were found"}


oc get AgentClusterInstall -n elvis2 elvis2 -o=custom-columns='STATUS:status.conditions[-3].message' 
STATUS
The installation is in progress: Finalizing cluster installation

curl -k https://assisted-service-rhacm.apps.sasha-0.qe.lab.redhat.com/api/assisted-install/v1/clusters/5dd0eff4-cf65-4425-a35e-583d5f40a5a5/logs?api_key=eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbHVzdGVyX2lkIjoiNWRkMGVmZjQtY2Y2NS00NDI1LWEzNWUtNTgzZDVmNDBhNWE1In0.uYTH7FbBvv0CSExYihj75UYoTUcFkWOTho5pHTjL0WC4KV86ubdts9ytDE92nOu_429bTpOJIXM81RMoGwHXXA
{"code":"500","href":"","id":500,"kind":"Error","reason":"No log files were found"}


oc get AgentClusterInstall -n elvis2 elvis2 -o=custom-columns='STATUS:status.conditions[-3].message' 
STATUS
The installation is in progress: Finalizing cluster installation. Cluster version status: progressing, message: Working towards 4.9.0-rc.1: 623 of 734 done (84% complete)

curl -k https://assisted-service-rhacm.apps.sasha-0.qe.lab.redhat.com/api/assisted-install/v1/clusters/5dd0eff4-cf65-4425-a35e-583d5f40a5a5/logs?api_key=eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbHVzdGVyX2lkIjoiNWRkMGVmZjQtY2Y2NS00NDI1LWEzNWUtNTgzZDVmNDBhNWE1In0.uYTH7FbBvv0CSExYihj75UYoTUcFkWOTho5pHTjL0WC4KV86ubdts9ytDE92nOu_429bTpOJIXM81RMoGwHXXA
{"code":"500","href":"","id":500,"kind":"Error","reason":"No log files were found"}




oc get AgentClusterInstall -n elvis2 elvis2 -o=custom-columns='STATUS:status.conditions[-3].message' 
STATUS
The installation is in progress: Finalizing cluster installation. Cluster version status: progressing, message: Working towards 4.9.0-rc.1: 629 of 734 done (85% complete)

curl -k https://assisted-service-rhacm.apps.sasha-0.qe.lab.redhat.com/api/assisted-install/v1/clusters/5dd0eff4-cf65-4425-a35e-583d5f40a5a5/logs?api_key=eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbHVzdGVyX2lkIjoiNWRkMGVmZjQtY2Y2NS00NDI1LWEzNWUtNTgzZDVmNDBhNWE1In0.uYTH7FbBvv0CSExYihj75UYoTUcFkWOTho5pHTjL0WC4KV86ubdts9ytDE92nOu_429bTpOJIXM81RMoGwHXXA
Warning: Binary output can mess up your terminal. Use "--output -" to tell 
Warning: curl to output it to your terminal anyway, or consider "--output 
Warning: <FILE>" to save to a file.

Sasha deployed the current ACM version, but we need to find out which assisted-service got deployed.
It should be assisted with branch release-ocm-2.4, we are double checking that.

Reproduced with quay.io/acm-d/acm-custom-registry:2.4.0-DOWNSTREAM-2021-10-01-17-28-41
Looking in:
https://github.com/open-cluster-management/pipeline/blob/2.4-integration/snapshots/downstream-2021-10-01-17-28-41-2.4.0.json#L83

Says: ocm 2.4

FailedQA


Version:
ACM: quay.io/acm-d/acm-custom-registry:v2.4.2-RC5
OCP: 4.10.0-rc.6



The cluster deployment was in the following state:
oc get AgentClusterInstall -n qe1 qe1 -o=custom-columns='STATUS:status.conditions[-3].message'
STATUS
The installation is in progress: Cluster has hosts pending user action



Came to debug it and wanted to collect the logs from agentclusterinstal

oc get agentclusterinstall qe1 -o json|jq ".status.debugInfo.logsURL" -r
https://assisted-service-rhacm.apps.rhos-qe.e2e.bos.redhat.com/api/assisted-install/v2/clusters/2eed5655-9860-4c9b-8b45-ad975cfabf85/logs?api_key=eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbHVzdGVyX2lkIjoiMmVlZDU2NTUtOTg2MC00YzliLThiNDUtYWQ5NzVjZmFiZjg1In0.XrSog3XyRvNnwAIdI6MS18c97WbJSkB1oKjwda13cik2etLKVIuTf17oAVLdsZohYXGdhDa-M0JCe17xLEU8og


curl -k -L https://assisted-service-rhacm.apps.rhos-qe.e2e.bos.redhat.com/api/assisted-install/v2/clusters/2eed5655-9860-4c9b-8b45-ad975cfabf85/logs?api_key=eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbHVzdGVyX2lkIjoiMmVlZDU2NTUtOTg2MC00YzliLThiNDUtYWQ5NzVjZmFiZjg1In0.XrSog3XyRvNnwAIdI6MS18c97WbJSkB1oKjwda13cik2etLKVIuTf17oAVLdsZohYXGdhDa-M0JCe17xLEU8og
{"code":"500","href":"","id":500,"kind":"Error","reason":"No log files were found"}[root@sealusa34 ~]#

Hi Alexander,

I see that you tried to verify this with ACM 2.4, yet the target release for this bug is ACM 2.5.
Is it possible that the version you tested against does not include the fix?

Oh I see.
I guess we need to set the fixed in version field.
I don't see the target release among flags. :)

The Target Release field indicates where the bug was fixed.  You can't see that field?

Per recent comments, I am setting this back to ON_QA.
Once QE confirms the fix, we should decide if we need this to land in 2.4 as well.

The Log Url is actually empty now:


 oc describe agentclusterinstalls.extensions.hive.openshift.io elvis2|grep  Log -B1 -A1

    Events URL:  https://assisted-service-multicluster-engine.apps.ocp-ztp-factory-hub-0.qe.lab.redhat.com/api/assisted-install/v2/events?api_key=eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbHVzdGVyX2lkIjoiNjc1MmRjYzYtMmY5MC00ZWVmLWEzYTMtNTA5ODY3ZjBjOTgzIn0.Z-2pPj61msJw5tm9KnP-ramMoTmf-W5eWqS8e4OvpGtjGv_vfd0dra9aOa-stg1daSLJMgcNyjO84DhT3yl4Sg&cluster_id=6752dcc6-2f90-4eef-a3a3-509867f0c983
    Logs URL:    
    State:       installing


oc get agentclusterinstall elvis2 -o json|jq ".status.conditions[-3].message"
"The installation is in progress: Installation in progress"

It depend on the stage of the installation, we are adding the logs only after they are actually collected, otherwise the user will have a link that don't download anything or just return an error.

 oc describe agentclusterinstalls.extensions.hive.openshift.io elvis2|grep  Log -B1 -A1;
    Events URL:  https://assisted-service-multicluster-engine.apps.ocp-ztp-factory-hub-0.qe.lab.redhat.com/api/assisted-install/v2/events?api_key=eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbHVzdGVyX2lkIjoiMDkyZjZkNGItMzg0ZC00OTU0LTkxOWEtZmE1ZjZkZjQ3Zjk0In0.IbJWoHQ4ow9mPh5foZpSNx-RmODM1SPnT9CegTmNfiO8fB-Hx9cDps6Zm5ZPKIFY1QdIADGgRAitITe38tE64A&cluster_id=092f6d4b-384d-4954-919a-fa5f6df47f94
    Logs URL:    https://assisted-service-multicluster-engine.apps.ocp-ztp-factory-hub-0.qe.lab.redhat.com/api/assisted-install/v2/clusters/092f6d4b-384d-4954-919a-fa5f6df47f94/logs?api_key=eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbHVzdGVyX2lkIjoiMDkyZjZkNGItMzg0ZC00OTU0LTkxOWEtZmE1ZjZkZjQ3Zjk0In0.hHtpc1ptaFPY9nz9rJtmVTSFv0jKgG7yoF8nEg5CjRy072qrmw4mDx7oigURyDmhsAdOPhMaq6wt92BiMstu9Q
    State:       adding-hosts


 oc get agentclusterinstall elvis2 -o json|jq ".status.conditions[-3].message"
"The installation has completed: Cluster is installed"

curl -k 'https://assisted-service-multicluster-engine.apps.ocp-ztp-factory-hub-0.qe.lab.redhat.com/api/assisted-install/v2/clusters/092f6d4b-384d-4954-919a-fa5f6df47f94/logs?api_key=eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbHVzdGVyX2lkIjoiMDkyZjZkNGItMzg0ZC00OTU0LTkxOWEtZmE1ZjZkZjQ3Zjk0In0.hHtpc1ptaFPY9nz9rJtmVTSFv0jKgG7yoF8nEg5CjRy072qrmw4mDx7oigURyDmhsAdOPhMaq6wt92BiMstu9Q'
Warning: Binary output can mess up your terminal. Use "--output -" to tell 
Warning: curl to output it to your terminal anyway, or consider "--output 
Warning: <FILE>" to save to a file.