Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1868144

Summary: [Assisted-4.5-M2] make deploy-all failed with Error from server (Forbidden): ingresscontrollers.operator.openshift.io is forbidden
Product: OpenShift Container Platform Reporter: Yuri Obshansky <yobshans>
Component: assisted-installerAssignee: Michael Filanov <mfilanov>
assisted-installer sub component: assisted-service QA Contact: Yuri Obshansky <yobshans>
Status: CLOSED NOTABUG Docs Contact:
Severity: high    
Priority: unspecified CC: aos-bugs
Version: 4.5   
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-08-17 12:25:31 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
PVC screenshot none

Description Yuri Obshansky 2020-08-11 19:41:40 UTC
Description of problem:
Deployment on QE PSI instance failed with error:
Error from server (Forbidden): ingresscontrollers.operator.openshift.io is forbidden: User "yobshans" cannot list resource "ingresscontrollers" in API group "operator.openshift.io" in the namespace "openshift-ingress-operator"

Deployment start using user: yobshans on https://console-openshift-console.apps.ocp.prod.psi.redhat.com/k8s/cluster/projects/ocp-edge-qe

See below full output

[root@seal12 assisted-service]# make deploy-all TARGET=oc-ingress APPLY_NAMESPACE=False NAMESPACE=ocp-edge-qe INGRESS_DOMAIN=edge-qe.apps.ocp.prod.psi.redhat.com
mkdir -p /home/assisted/assisted-service/build
python3 ./tools/deploy_namespace.py --deploy-namespace False --namespace "ocp-edge-qe"
Not deploying namespace
python3 ./tools/deploy_postgres.py --namespace "ocp-edge-qe"
Deploying /home/assisted/assisted-service/build/postgres-secret.yaml
secret/assisted-installer-rds configured

Deploying /home/assisted/assisted-service/build/postgres-deployment.yaml
deployment.apps/postgres created
service/postgres created

Error from server (Forbidden): persistentvolumeclaims "postgres-pv-claim" is forbidden: User "yobshans" cannot get resource "persistentvolumeclaims" in API group "" in the namespace "assisted-installer"
Using default size 10Gi
Deploying /home/assisted/assisted-service/build/postgres-storage.yaml
persistentvolumeclaim/postgres-pv-claim unchanged

python3 ./tools/deploy_s3.py --namespace "ocp-edge-qe"
Deploying /home/assisted/assisted-service/build/scality-deployment.yaml
deployment.apps/scality created
service/cloudserver-front created

Deploying /home/assisted/assisted-service/build/scality-storage.yaml
persistentvolumeclaim/scality-pv-claim unchanged

sleep 5;  # wait for service to get an address
make deploy-s3-secret
make[1]: Entering directory '/home/assisted/assisted-service'
python3 ./tools/deploy_scality_configmap.py --namespace "ocp-edge-qe"
Deploying /home/assisted/assisted-service/build/scality-secret.yaml
secret/assisted-installer-s3 configured

make[1]: Leaving directory '/home/assisted/assisted-service'
python3 ./tools/deploy_route53.py --secret "" --namespace "ocp-edge-qe"
python3 ./tools/deploy_inventory_service.py --target "oc-ingress" --domain "edge-qe.apps.ocp.prod.psi.redhat.com" --namespace "ocp-edge-qe"
Deploying /home/assisted/assisted-service/build/assisted-service-service.yaml
service/assisted-service created

Deploying /home/assisted/assisted-service/build/assisted-installer-ingress.yaml
ingress.networking.k8s.io/assisted-installer unchanged

sleep 5;  # wait for service to get an address
python3 ./tools/deploy_assisted_installer_configmap.py --target "oc-ingress" --domain "edge-qe.apps.ocp.prod.psi.redhat.com" --base-dns-domains "" --namespace "ocp-edge-qe"  --enable-auth "False"
Deploying /home/assisted/assisted-service/build/assisted-service-configmap.yaml
installer-image-build image for deployment: quay.io/ocpmetal/installer-image-build:latest
agent image for deployment: quay.io/ocpmetal/agent:latest
ignition-manifests-and-kubeconfig-generate image for deployment: quay.io/ocpmetal/ignition-manifests-and-kubeconfig-generate:latest
assisted-installer image for deployment: quay.io/ocpmetal/assisted-installer:latest
assisted-installer-controller image for deployment: quay.io/ocpmetal/assisted-installer-controller:latest
connectivity_check image for deployment: quay.io/ocpmetal/connectivity_check:latest
inventory image for deployment: quay.io/ocpmetal/inventory:latest
assisted-service image for deployment: quay.io/ocpmetal/assisted-service:latest
configmap/assisted-service-config created

python3 ./tools/deploy_role.py --namespace "ocp-edge-qe"
Deploying /home/assisted/assisted-service/build/default_role.yaml
serviceaccount/default unchanged
role.rbac.authorization.k8s.io/default unchanged
rolebinding.rbac.authorization.k8s.io/default unchanged

python3 ./tools/deploy_assisted_installer.py  --namespace "ocp-edge-qe" 
assisted-service image for deployment: quay.io/ocpmetal/assisted-service:latest
Deploying /home/assisted/assisted-service/build/assisted-service.yaml
deployment.apps/assisted-service created

python3 ./tools/wait_for_assisted_service.py --target oc-ingress --namespace "ocp-edge-qe"
I0811 21:04:54.550036  168990 request.go:621] Throttling request took 1.191255144s, request: GET:https://api.ocp.prod.psi.redhat.com:6443/apis/serving.knative.dev/v1beta1?timeout=32s
Error from server (Forbidden): ingresscontrollers.operator.openshift.io is forbidden: User "yobshans" cannot list resource "ingresscontrollers" in API group "operator.openshift.io" in the namespace "openshift-ingress-operator"
Traceback (most recent call last):
  File "./tools/wait_for_assisted_service.py", line 44, in <module>
    main()
  File "./tools/wait_for_assisted_service.py", line 33, in main
    service_url = utils.get_service_url(SERVICE, deploy_options.target, deploy_options.domain, deploy_options.namespace)
  File "/home/assisted/assisted-service/tools/utils.py", line 44, in get_service_url
    service_host = f"{service}.{get_domain(domain)}"
  File "/home/assisted/assisted-service/tools/utils.py", line 60, in get_domain
    return check_output(cmd).split()[-1]
  File "/home/assisted/assisted-service/tools/utils.py", line 15, in check_output
    return subprocess.check_output(cmd, shell=True).decode("utf-8")
  File "/usr/lib64/python3.6/subprocess.py", line 356, in check_output
    **kwargs).stdout
  File "/usr/lib64/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'kubectl -n assisted-installer get ingresscontrollers.operator.openshift.io -n openshift-ingress-operator -o custom-columns=:.status.domain' returned non-zero exit status 1.
make: *** [Makefile:138: deploy-service] Error 1
[root@seal12 assisted-service]# 

https://github.com/openshift/assisted-service/blob/6af4f3238b10a2cbc628f48e26af68b07e189b34/tools/utils.py#L41

Version-Release number of selected component (if applicable):
latest from August 11

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Michael Filanov 2020-08-12 07:25:07 UTC
You don't have permissions to create postgres-pv-claim in this namespace, It's not related to the deployment script. need to contact someone from PSI to get permissions.

Comment 2 Yuri Obshansky 2020-08-12 12:53:36 UTC
(In reply to Michael Filanov from comment #1)
> You don't have permissions to create postgres-pv-claim in this namespace,
> It's not related to the deployment script. need to contact someone from PSI
> to get permissions.

It is not related to postgres-pv-claim error
Error about namespace "assisted-installer" while I use namespace "ocp-edge-qe".
"Error from server (Forbidden): persistentvolumeclaims "postgres-pv-claim" is forbidden: User "yobshans" cannot get resource "persistentvolumeclaims" in API group "" in the namespace "assisted-installer"
Some hardcoded string in deploy_postgres.py --namespace "ocp-edge-qe" 

Also Persistent Volume Claims created successfully -> see attached screenshot.

The problem with Command 'kubectl -n assisted-installer get ingresscontrollers.operator.openshift.io -n openshift-ingress-operator -o custom-columns=:.status.domain'
which I did not find in code.

Comment 3 Yuri Obshansky 2020-08-12 12:54:03 UTC
Created attachment 1711181 [details]
PVC screenshot

Comment 4 Yuri Obshansky 2020-08-12 13:01:40 UTC
More information about deployment status

[root@seal12 assisted-service]# oc get pods -n ocp-edge-qe
NAME                                                              READY   STATUS      RESTARTS   AGE
assisted-service-d9f59598-kvnp7                                   1/1     Running     3          7m34s
dummyimage-00000000-0000-0000-0000-000000000000-20200812125rqc8   0/1     Completed   0          6m16s
ocp-metal-ui-6965f6f467-9c79g                                     1/1     Running     0          11s
postgres-7b556f9f86-x98k6                                         1/1     Running     0          8m3s
scality-7689b7d46c-bvw89                                          1/1     Running     0          7m59s

[root@seal12 assisted-service]# oc get deployments -n ocp-edge-qe
NAME               READY   UP-TO-DATE   AVAILABLE   AGE
assisted-service   1/1     1            1           9m47s
ocp-metal-ui       1/1     1            1           2m24s
postgres           1/1     1            1           10m
scality            1/1     1            1           10m

[root@seal12 assisted-service]# oc get pvc -n ocp-edge-qe
NAME                STATUS   VOLUME           CAPACITY   ACCESS MODES   STORAGECLASS   AGE
postgres-pv-claim   Bound    csi-155253b032   10Gi       RWO            dynamic-nfs    8m27s
scality-pv-claim    Bound    csi-6d469ad87c   20Gi       RWO            dynamic-nfs    8m23s

[root@seal12 assisted-service]# oc get secrets -n ocp-edge-qe
NAME                       TYPE                                  DATA   AGE
assisted-installer-rds     Opaque                                5      10m
assisted-installer-s3      Opaque                                7      10m
builder-dockercfg-nwfm5    kubernetes.io/dockercfg               1      20d
builder-token-6xvpx        kubernetes.io/service-account-token   4      20d
builder-token-gs2nb        kubernetes.io/service-account-token   4      20d
default-dockercfg-srggv    kubernetes.io/dockercfg               1      20d
default-token-7p9lp        kubernetes.io/service-account-token   4      20d
default-token-d6qj6        kubernetes.io/service-account-token   4      20d
deployer-dockercfg-gllhl   kubernetes.io/dockercfg               1      20d
deployer-token-c5zzd       kubernetes.io/service-account-token   4      20d
deployer-token-fxnvs       kubernetes.io/service-account-token   4      20d
pipeline-dockercfg-dpl2h   kubernetes.io/dockercfg               1      20d
pipeline-token-7sxlc       kubernetes.io/service-account-token   4      20d
pipeline-token-ljwkt       kubernetes.io/service-account-token   4      20d

[root@seal12 assisted-service]# oc get configmaps -n ocp-edge-qe
NAME                      DATA   AGE
assisted-service-config   15     10m
ocp-metal-ui              1      3m25s

[root@seal12 assisted-service]# oc get services -n ocp-edge-qe
NAME                TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
assisted-service    LoadBalancer   172.30.58.73    <pending>     8090:30209/TCP   11m
cloudserver-front   ClusterIP      None            <none>        8000/TCP         11m
ocp-metal-ui        LoadBalancer   172.30.82.185   <pending>     80:30049/TCP     4m5s
postgres            LoadBalancer   172.30.38.151   <pending>     5432:30499/TCP   11m

NAME                          HOST/PORT                                                    PATH   SERVICES           PORT   TERMINATION   WILDCARD
assisted-installer-9nbkz      assisted-installer.edge-qe.apps.ocp.prod.psi.redhat.com             assisted-service   8090                 None
assisted-installer-ui-nvlw6   assisted-installer-ui.edge-qe.apps.ocp.prod.psi.redhat.com          ocp-metal-ui       8080                 None

NAME                    HOSTS                                                        ADDRESS   PORTS   AGE
assisted-installer      assisted-installer.edge-qe.apps.ocp.prod.psi.redhat.com                80      2d21h
assisted-installer-ui   assisted-installer-ui.edge-qe.apps.ocp.prod.psi.redhat.com             80      2d21h

Comment 5 Michael Filanov 2020-08-17 12:25:31 UTC
duplicates https://bugzilla.redhat.com/show_bug.cgi?id=1868468