**What happened**: Configuring Submariner Addon on a GCP managed cluster, that was imported externally (not created within ACM), failed on error x509: certificate specifies an incompatible key usage: https://qe-jenkins-csb-skynet.apps.ocp-c1.prod.psi.redhat.com/job/ACM-2.6.2-Submariner-0.13.1-AWS-GCP-Globalnet/46/Test-Report/ SubmarinerConfig resource created indicates that the gateway label could not be created: status: conditions: - lastTransitionTime: "2022-10-11T20:29:28Z" message: SubmarinerConfig was applied reason: SubmarinerConfigApplied status: "True" type: SubmarinerConfigApplied - lastTransitionTime: "2022-10-11T20:29:39Z" message: 'Failed to prepare submariner cluster environment: error creating firewall rule "gcp-nmanos-cluster-c1-6tfr2-submariner-public-ports-ingress": error retrieving firewall rule "gcp-nmanos-cluster-c1-6tfr2-submariner-public-ports-ingress": Get "https://compute.googleapis.com/compute/v1/projects/gc-acm-dev/global/firewalls/gcp-nmanos-cluster-c1-6tfr2-submariner-public-ports-ingress?alt=json&prettyPrint=false": oauth2: cannot fetch token: Post "https://oauth2.googleapis.com/token": x509: certificate specifies an incompatible key usage' reason: SubmarinerClusterEnvPreparationFailed status: "False" type: SubmarinerClusterEnvironmentPrepared **What you expected to happen**: Submariner Addon should be configured, and Gateway node should be created on the managed GCP cluster. **How to reproduce it (as minimally and precisely as possible)**: 1. Install OCP 4.10 on GCP (not via ACM) 2. Import the GCP cluster into ACM managed clusters, and add it to a cluster-set. 3. Configure Submariner Addon on this cluster set. **Anything else we need to know?**: Note that in ACM 2.5.z recent releases, with same OCP version for the GCP platform, this error does NOT occur: https://qe-jenkins-csb-skynet.apps.ocp-c1.prod.psi.redhat.com/job/ACM-2.5.3-Submariner-0.12.2-AWS-GCP-Globalnet/6/Test-Report/ I.e. It's a unique issue for ACM 2.6 Submariner Addon. **Environment**: AWS ACM Hub: - OCP version: 4.11.8 Pull From: quay.io/openshift-release-dev/ocp-release@sha256:6499bc69a0707fcad481c3cb73226c364586761c45b867dc31b345c6e6204e28 - kubernetes 1.24.0 GCP Managed Cluster - OCP version: 4.10.0 Pull From: quay.io/openshift-release-dev/ocp-release@sha256:a63c470411c087c0568729fa5faf32cd8d5fa1db3c73e28f3989f17f5c458351- - kubernetes 1.23.3 - Submariner version: 0.13.1 Submariner images: https://access.redhat.com/containers/#/registry.access.redhat.com/openshift/ose-operator-registry/images/v4.11.0-202209212107.p0.g6094188.assembly.stream https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2/submariner-operator-bundle/images/v0.13.1-3 https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2/submariner-operator-bundle/images/v0.13.1-3 https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2/lighthouse-agent-rhel8/images/v0.13.1-4 https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2/lighthouse-coredns-rhel8/images/v0.13.1-3 https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2/submariner-gateway-rhel8/images/v0.13.1-4 https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2/submariner-globalnet-rhel8/images/v0.13.1-4 https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2/submariner-rhel8-operator/images/v0.13.1-4 https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2/submariner-route-agent-rhel8/images/v0.13.1-4 https://access.redhat.com/containers/#/registry.access.redhat.com/openshift/ose-operator-registry/images/v4.11.0-202209212107.p0.g6094188.assembly.stream https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2/acm-operator-bundle/images/v2.6.2-33
Created attachment 1917946 [details] ACM UI: Submariner Addon nodes not labeled
Created attachment 1917947 [details] ACM UI: Submariner Cluster Set
Please see K8S issue that is related to "x509: certificate specifies an incompatible key usage": https://github.com/kubernetes/kops/issues/2354 Is there an option in ACM (or in OCP), to set the managed cluster certificate usage type with "client auth", when creating the secret ? For example, can we set it like this: apiVersion: certificates.k8s.io/v1beta1 kind: CertificateSigningRequest metadata: name: $CERTIFICATE_NAME spec: groups: - system:authenticated request: $(cat $CSR_FILE | base64 | tr -d '\n') usages: - digital signature - key encipherment - client auth
Hi Aswin, 1. Have you had a chance to look at this issue? Do you know why we are only seeing it now, i.e was there any recent change of behavior in Submariner 0.13.1 or ACM 2.6.2? 2. Does this qualify as a Sev-1 (urgent stop ship) issue or is there a workaround that can be applied? Thanks, Nir
@nyechiel This issue seems to be not affecting all the cluster, when tried with upstream build it seems to be working fine. As Noam pointed out it seems to be not affecting newly created clusters too. If we try to remove an imported cluster GCP in ACM it fails due to the same error. So it is not just Submariner that is affected. It does not seems to be an issue due to a recent change in Submariner but we are yet to identify the root cause, . There is no workaround other than manually running cloud prepare using subctl which seems to work fine.
Not sure if it's the same root cause, but I see a similar issue was reported in FIPS env' while trying to access compute.googleapis.com: https://bugzilla.redhat.com/show_bug.cgi?id=2133019#c1
Testing same scenario (same OCP, ACM and Submariner versions), only this time on GCP cluster without FIPS (openshift installer yaml with "fips: false") - then SubmarinerConfig completed successfully: $ oc describe managedclusteraddons "submariner" -n "acm-gcp-nmanos-c1" Name: submariner Namespace: acm-gcp-nmanos-c1 Labels: <none> Annotations: <none> API Version: addon.open-cluster-management.io/v1alpha1 Kind: ManagedClusterAddOn Metadata: Creation Timestamp: 2022-10-19T18:17:02Z Finalizers: submarineraddon.open-cluster-management.io/submariner-addon-cleanup submarineraddon.open-cluster-management.io/submariner-addon-cleanup Generation: 1 Resource Version: 219555 UID: d1f9ea61-5955-4519-9607-07f7e385ae4f Spec: Install Namespace: submariner-operator Status: Add On Configuration: Cr Name: Crd Name: Add On Meta: Description: Display Name: Conditions: Last Transition Time: 2022-10-19T18:17:02Z Message: Registration of the addon agent is configured Reason: RegistrationConfigured Status: True Type: RegistrationApplied Last Transition Time: 2022-10-19T18:17:03Z Message: manifest of addon applied successfully Reason: ManifestApplied Status: True Type: ManifestApplied Last Transition Time: 2022-10-19T18:17:02Z Message: client certificate rotated starting from 2022-10-19 18:12:02 +0000 UTC to 2022-10-20 11:14:26 +0000 UTC Reason: ClientCertificateUpdated Status: True Type: ClusterCertificateRotated Last Transition Time: 2022-10-19T18:17:03Z Message: Reason: BrokerConfigApplied Status: True Type: SubmarinerBrokerConfigApplied Last Transition Time: 2022-10-19T18:17:35Z Message: submariner add-on is available. Reason: ManagedClusterAddOnLeaseUpdated Status: True Type: Available Last Transition Time: 2022-10-19T18:19:39Z Message: The nodes "gcp-nmanos-c1-5msz8-submariner-gw-us-east1-b-jwkml" are labeled with "submariner.io/gateway" Reason: SubmarinerGatewayNodesLabeled Status: True Type: SubmarinerGatewayNodesLabeled Last Transition Time: 2022-10-19T18:20:09Z Message: Submariner (submariner.v0.13.1) is deployed on managed cluster. Reason: SubmarinerAgentDeployed Status: False Type: SubmarinerAgentDegraded Last Transition Time: 2022-10-19T18:17:37Z Message: There are no connections on gateways Reason: ConnectionsNotEstablished Status: True Type: SubmarinerConnectionDegraded Health Check: Mode: Lease Registrations: Signer Name: kubernetes.io/kube-apiserver-client Subject: Groups: system:open-cluster-management:cluster:acm-gcp-nmanos-c1:addon:submariner system:open-cluster-management:addon:submariner system:authenticated User: system:open-cluster-management:cluster:acm-gcp-nmanos-c1:addon:submariner:agent:submariner-addon-agent Events: <none>
*** This bug has been marked as a duplicate of bug 2133019 ***
This issue is also relevant for ACM 2.5.6 when deploying submariner operator on GCP cluster with FIPS: https://qe-jenkins-csb-skynet.apps.ocp-c1.prod.psi.redhat.com/view/ACM%202.5/job/ACM-2.5.6-Submariner-0.12.3-AWS-GCP-Globalnet/43/Test-Report/ oc describe deployments -n "submariner-operator" Name: submariner-addon Namespace: submariner-operator CreationTimestamp: Thu, 15 Dec 2022 04:47:45 +0200 Labels: app=submariner-addon Annotations: deployment.kubernetes.io/revision: 1 Selector: app=submariner-addon Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable StrategyType: RollingUpdate MinReadySeconds: 0 RollingUpdateStrategy: 25% max unavailable, 25% max surge Pod Template: Labels: app=submariner-addon Service Account: submariner-addon-sa Containers: submariner-addon: Image: registry.redhat.io/rhacm2/submariner-addon-rhel8@sha256:d7f416b538d07104b351039c096d2677488fe161270405ab941f6f5f97431002 Port: <none> Host Port: <none> Args: /submariner agent --hub-kubeconfig=/var/run/hub/kubeconfig --cluster-name=acm-gcp-nmanos-c1 Environment: <none> Mounts: /var/run/hub from hub-config (rw) Volumes: hub-config: Type: Secret (a volume populated by a Secret) SecretName: submariner-hub-kubeconfig Optional: false Conditions: Type Status Reason ---- ------ ------ Progressing True NewReplicaSetAvailable Available True MinimumReplicasAvailable OldReplicaSets: <none> NewReplicaSet: submariner-addon-6d94d476b4 (1/1 replicas created) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ScalingReplicaSet 23m deployment-controller Scaled up replica set submariner-addon-6d94d476b4 to 1 Warning FastControllerResync 23m submariner-agent-submarineragentconfigcontroller Controller "SubmarinerAgentConfigController" resync interval is set to 0s which might lead to client request throttling Warning FastControllerResync 23m submariner-agent-submarineragentstatuscontroller Controller "SubmarinerAgentStatusController" resync interval is set to 0s which might lead to client request throttling Warning FastControllerResync 23m submariner-agent-submarineragentstatuscontroller Controller "SubmarinerAgentStatusController" resync interval is set to 0s which might lead to client request throttling Warning FastControllerResync 23m submariner-agent-submarinerconnectionsstatuscontroller Controller "SubmarinerConnectionsStatusController" resync interval is set to 0s which might lead to client request throttling Normal ManagedClusterAddOnStatusUpdated 23m submariner-agent-submarineragentstatuscontroller Updated status conditions: []v1.Condition{v1.Condition{Type:"RegistrationApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2022, time.December, 15, 2, 47, 45, 0, time.Local), Reason:"RegistrationConfigured", Message:"Registration of the addon agent is configured"}, v1.Condition{Type:"ManifestApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2022, time.December, 15, 2, 47, 45, 0, time.Local), Reason:"AddonManifestApplied", Message:"manifest of addon applied successfully"}, v1.Condition{Type:"SubmarinerBrokerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2022, time.December, 15, 2, 47, 45, 0, time.Local), Reason:"BrokerConfigApplied", Message:""}, v1.Condition{Type:"SubmarinerAgentDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2022, time.December, 15, 2, 47, 53, 0, time.Local), Reason:"CSVNotInstalled,NoOperatorDeployment,NoGatewayDaemonSet,NoRouteAgentDaemonSet", Message:"The submariner-operator CSV (submariner.v0.12) is not installed from channel (stable-0.12) in catalog source (submariner-operator/submariner-stable-0-12-catalog)\nThe submariner operator deployment does not exist\nThe gateway daemon set does not exist\nThe route agents are not found"}} Normal ManagedClusterAddOnStatusUpdated 23m submariner-agent-submarineragentstatuscontroller Updated status conditions: []v1.Condition{v1.Condition{Type:"RegistrationApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2022, time.December, 15, 2, 47, 45, 0, time.Local), Reason:"RegistrationConfigured", Message:"Registration of the addon agent is configured"}, v1.Condition{Type:"ManifestApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2022, time.December, 15, 2, 47, 45, 0, time.Local), Reason:"AddonManifestApplied", Message:"manifest of addon applied successfully"}, v1.Condition{Type:"SubmarinerBrokerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2022, time.December, 15, 2, 47, 45, 0, time.Local), Reason:"BrokerConfigApplied", Message:""}, v1.Condition{Type:"SubmarinerAgentDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2022, time.December, 15, 2, 47, 53, 0, time.Local), Reason:"CSVNotInstalled,NoOperatorDeployment,NoGatewayDaemonSet,NoRouteAgentDaemonSet", Message:"The submariner-operator CSV (submariner.v0.12) is not installed from channel (stable-0.12) in catalog source (submariner-operator/submariner-stable-0-12-catalog)\nThe submariner operator deployment does not exist\nThe gateway daemon set does not exist\nThe route agents are not found"}, v1.Condition{Type:"SubmarinerGatewayNodesLabeled", Status:"False", ObservedGeneration:0, LastTransitionTime:time.Date(2022, time.December, 15, 2, 47, 54, 0, time.Local), Reason:"SubmarinerGatewayNodesUnlabeled", Message:"There are no nodes with label \"submariner.io/gateway\""}} Normal GCPCloudProvider 23m submariner-agent-submarineragentconfigcontroller Deploying dedicated gateway node in zone "us-east1-d" Normal GCPCloudProvider 23m submariner-agent-submarineragentconfigcontroller Successfully deployed gateway node Normal SubmarinerConfigStatusUpdated 23m submariner-agent-submarineragentconfigcontroller Updated status conditions: []v1.Condition{v1.Condition{Type:"SubmarinerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2022, time.December, 15, 2, 47, 45, 0, time.Local), Reason:"SubmarinerConfigApplied", Message:"SubmarinerConfig was applied"}, v1.Condition{Type:"SubmarinerClusterEnvironmentPrepared", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2022, time.December, 15, 2, 47, 56, 0, time.Local), Reason:"SubmarinerClusterEnvPrepared", Message:"Submariner cluster environment was prepared"}, v1.Condition{Type:"SubmarinerGatewaysLabeled", Status:"False", ObservedGeneration:0, LastTransitionTime:time.Date(2022, time.December, 15, 2, 47, 56, 0, time.Local), Reason:"InsufficientNodes", Message:"The 0 worker nodes labeled as gateways (\"\") does not match the desired number 1"}} Warning GCPCloudProvider 23m submariner-agent-submarineragentconfigcontroller Failed [error creating firewall rule "gcp-nmanos-c1-w928m-submariner-public-ports-ingress": error updating firewall rule &compute.Firewall{Allowed:[]*compute.FirewallAllowed{(*compute.FirewallAllowed)(0xc00070ff20), (*compute.FirewallAllowed)(0xc0004d0000), (*compute.FirewallAllowed)(0xc0004d0060), (*compute.FirewallAllowed)(0xc0004d0120)}, CreationTimestamp:"", Denied:[]*compute.FirewallDenied(nil), Description:"", DestinationRanges:[]string(nil), Direction:"INGRESS", Disabled:false, Id:0x0, Kind:"", LogConfig:(*compute.FirewallLogConfig)(nil), Name:"gcp-nmanos-c1-w928m-submariner-public-ports-ingress", Network:"projects/gc-acm-dev/global/networks/gcp-nmanos-c1-w928m-network", Priority:0, SelfLink:"", SourceRanges:[]string(nil), SourceServiceAccounts:[]string(nil), SourceTags:[]string(nil), TargetServiceAccounts:[]string(nil), TargetTags:[]string{"submariner-io-gateway-node"}, ServerResponse:googleapi.ServerResponse{HTTPStatusCode:0, Header:http.Header(nil)}, ForceSendFields:[]string(nil), NullFields:[]string(nil)}: googleapi: Error 400: The resource 'projects/gc-acm-dev/global/firewalls/gcp-nmanos-c1-w928m-submariner-public-ports-ingress' is not ready, resourceNotReady]
This bug seems to be resolved with recent OCP and ACM releases! Using OCP 4.10.47 with ACM 2.6.4-15 and Submariner 0.13.3-4 the FIPS error was not reproduced: https://qe-jenkins-csb-skynet.apps.ocp-c1.prod.psi.redhat.com/job/ACM-2.6.4-Submariner-0.13.3-AWS-GCP-Globalnet/19/Test-Report Polarion results: Test suite "ACM-2.6.4-Submariner-0.13.3-AWS-GCP-Globalnet_sys" contains 69 test cases: https://polarion.engineering.redhat.com/polarion/#/project/RHACM4K/testrun?id=multi-cluster-networking_acm-2-6-4-submariner-0-13-3-aws-gcp-globalnet-sys Test suite "ACM-2.6.4-Submariner-0.13.3-AWS-GCP-Globalnet_e2e" contains 29 test cases: https://polarion.engineering.redhat.com/polarion/#/project/RHACM4K/testrun?id=multi-cluster-networking_acm-2-6-4-submariner-0-13-3-aws-gcp-globalnet-e2e Test suite "ACM-2.6.4-Submariner-0.13.3-AWS-GCP-Globalnet_lighthouse" contains 16 test cases: https://polarion.engineering.redhat.com/polarion/#/project/RHACM4K/testrun?id=multi-cluster-networking_acm-2-6-4-submariner-0-13-3-aws-gcp-globalnet-lighthouse The related bug https://bugzilla.redhat.com/show_bug.cgi?id=2133019 was also fixed and verified recently.