Bug 1704914
| Summary: | SSP operator failed to deploy | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Container Native Virtualization (CNV) | Reporter: | Israel Pinto <ipinto> | ||||||
| Component: | Installation | Assignee: | Martin Sivák <msivak> | ||||||
| Status: | CLOSED ERRATA | QA Contact: | Irina Gulina <igulina> | ||||||
| Severity: | urgent | Docs Contact: | |||||||
| Priority: | high | ||||||||
| Version: | 2.0 | CC: | cnv-qe-bugs, fdeutsch, gouyang, igulina, ncredi, rhallise, sgordon | ||||||
| Target Milestone: | --- | ||||||||
| Target Release: | 2.0 | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | hyperconverged-cluster-operator-container-v2.0.0-12 | Doc Type: | If docs needed, set a value | ||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2019-07-24 20:15:51 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Attachments: |
|
||||||||
Created attachment 1560492 [details]
hco-operator info
Failed in build hco-bundle-registry:v2.0.0-13 Check again once kubevirt-ssp-operator-2.0.0-13 is available in HCO please. We just fixed a bunch of issues this morning. the issue is fixed on hco:v2.0.0-13.
# oc get pod kubevirt-ssp-operator-578cb75cbd-g77lk -n kubevirt-hyperconverged
NAME READY STATUS RESTARTS AGE
kubevirt-ssp-operator-578cb75cbd-g77lk 1/1 Running 0 29m
# oc describe pod kubevirt-ssp-operator-578cb75cbd-g77lk -n kubevirt-hyperconverged
Name: kubevirt-ssp-operator-578cb75cbd-g77lk
Namespace: kubevirt-hyperconverged
Priority: 0
PriorityClassName: <none>
Node: working-lmsvw-worker-0-8jjnt/192.168.126.52
Start Time: Wed, 15 May 2019 23:54:15 -0400
Labels: name=kubevirt-ssp-operator
pod-template-hash=578cb75cbd
Annotations: alm-examples=[
{
"apiVersion": "hco.kubevirt.io/v1alpha1",
"kind": "HyperConverged",
"metadata": {
"name": "kubevirt-hyperconverged",
"namespace": "kubevirt-hyperconverged"
...
capabilities=Full Lifecycle
categories=OpenShift Optional
certified=false
containerImage=brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/hyperconverged-cluster-operator:v2.0.0-20
createdAt=2019-04-17 16:01:59
description=Creates and maintains a HyperConverged KubeVirt Deployment
k8s.v1.cni.cncf.io/networks-status=[{
"name": "openshift-sdn",
"interface": "eth0",
"ips": [
"10.130.0.40"
],
"default": true,
"dns": {}
}]
olm.operatorGroup=hco-operatorgroup
olm.operatorNamespace=kubevirt-hyperconverged
olm.targetNamespaces=
openshift.io/scc=anyuid
repository=https://github.com/kubevirt/hyperconverged-cluster-operator
support=false
Status: Running
IP: 10.130.0.40
Controlled By: ReplicaSet/kubevirt-ssp-operator-578cb75cbd
Containers:
kubevirt-ssp-operator:
Container ID: cri-o://666914acad673cdb897f1bee489a4fcaee83f552abd90eb27120774e898a34d8
Image: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/kubevirt-ssp-operator:v2.0.0-10
Image ID: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/kubevirt-ssp-operator@sha256:4b875c20180fc0fd7a8694e673140d3862113f15ee197aa4bdd6e434047086d1
Port: 60000/TCP
Host Port: 0/TCP
State: Running
Started: Wed, 15 May 2019 23:57:07 -0400
Ready: True
Restart Count: 0
Environment:
POD_NAME: kubevirt-ssp-operator-578cb75cbd-g77lk (v1:metadata.name)
WATCH_NAMESPACE: kubevirt-hyperconverged (v1:metadata.namespace)
OPERATOR_NAME: kubevirt-ssp-operator
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kubevirt-ssp-operator-token-m4k2x (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
kubevirt-ssp-operator-token-m4k2x:
Type: Secret (a volume populated by a Secret)
SecretName: kubevirt-ssp-operator-token-m4k2x
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 29m default-scheduler Successfully assigned kubevirt-hyperconverged/kubevirt-ssp-operator-578cb75cbd-g77lk to working-lmsvw-worker-0-8jjnt
Normal Pulling 29m kubelet, working-lmsvw-worker-0-8jjnt Pulling image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/kubevirt-ssp-operator:v2.0.0-10"
Normal Pulled 26m kubelet, working-lmsvw-worker-0-8jjnt Successfully pulled image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/kubevirt-ssp-operator:v2.0.0-10"
Normal Created 26m kubelet, working-lmsvw-worker-0-8jjnt Created container kubevirt-ssp-operator
Normal Started 26m kubelet, working-lmsvw-worker-0-8jjnt Started container kubevirt-ssp-operator
Which installation you used: hco-bundle-registry:v2.0.0-13? If yes this build is still broken and missing a lot of pods, better to test with working installation. Created attachment 1570053 [details]
ssp on hco-bundle-registry:v2.0.0-15
@Israel, yes, that was on hco-bundle-registry:v2.0.0-13 For hco-bundle-registry:v2.0.0-15 ssp works too. See the attachment. Will it be ok or do you want to wait for another build? Irina, Yes it not failing, what i want to verify that it contains all what it should, See https://bugzilla.redhat.com/show_bug.cgi?id=1709268#c10 looks like we need to verify the API also. About the common templates still missing but handles in diff BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1709268#c14 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:1850 |
Description of problem: On HCO deployment we failed to deploy SSP operator [root@dell-r640-008 ~]# oc describe pod kubevirt-ssp-operator-6f89bd7c46-pf956 -n kubevirt-hyperconverged Name: kubevirt-ssp-operator-6f89bd7c46-pf956 Namespace: kubevirt-hyperconverged Priority: 0 PriorityClassName: <none> Node: working-8mxhm-worker-0-7h55w/192.168.126.52 Start Time: Tue, 30 Apr 2019 17:51:37 +0300 Labels: name=kubevirt-ssp-operator pod-template-hash=6f89bd7c46 Annotations: alm-examples=[ { "apiVersion": "hco.kubevirt.io/v1alpha1", "kind": "HyperConverged", "metadata": { "name": "kubevirt-hyperconverged", "namespace": "kubevirt-hyperconverged" ... capabilities=Full Lifecycle categories=OpenShift Optional certified=false containerImage=brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/hyperconverged-cluster-operator:v2.0.0-9 createdAt=2019-04-17T16:01:59Z description=Creates and maintains a HyperConverged KubeVirt Deployment k8s.v1.cni.cncf.io/networks-status=[{ "name": "openshift-sdn", "interface": "eth0", "ips": [ "10.130.0.23" ], "default": true, "dns": {} }] olm.operatorGroup=hco-operatorgroup olm.operatorNamespace=kubevirt-hyperconverged olm.targetNamespaces= openshift.io/scc=anyuid repository=https://github.com/kubevirt/hyperconverged-cluster-operator support=false Status: Running IP: 10.130.0.23 Controlled By: ReplicaSet/kubevirt-ssp-operator-6f89bd7c46 Containers: kubevirt-ssp-operator: Container ID: cri-o://98f875cc21b2378a3036c362c660d3a331b33e5045feeaedd5b9d11359c0e1cc Image: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/kubevirt-ssp-operator:v2.0.0-7 Image ID: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/kubevirt-ssp-operator@sha256:fb56ca9ac65a69f7805455fe867989700872c42492e610cae616c55680a1f889 Port: 60000/TCP Host Port: 0/TCP State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Exit Code: 1 Started: Tue, 30 Apr 2019 22:12:59 +0300 Finished: Tue, 30 Apr 2019 22:13:00 +0300 Ready: False Restart Count: 55 Environment: WATCH_NAMESPACE: kubevirt-hyperconverged (v1:metadata.namespace) OPERATOR_NAME: kubevirt-ssp-operator Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kubevirt-ssp-operator-token-7qjf6 (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: kubevirt-ssp-operator-token-7qjf6: Type: Secret (a volume populated by a Secret) SecretName: kubevirt-ssp-operator-token-7qjf6 Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Pulled 41m (x48 over 4h) kubelet, working-8mxhm-worker-0-7h55w Successfully pulled image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/kubevirt-ssp-operator:v2.0.0-7" Normal Pulling 31m (x50 over 4h) kubelet, working-8mxhm-worker-0-7h55w Pulling image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/kubevirt-ssp-operator:v2.0.0-7" Warning BackOff 1m (x1169 over 4h) kubelet, working-8mxhm-worker-0-7h55w Back-off restarting failed container root@dell-r640-008 ~]# oc logs kubevirt-ssp-operator-6f89bd7c46-pf956 -n kubevirt-hyperconverged {"level":"info","ts":1556651267.9799497,"logger":"cmd","msg":"Go Version: go1.10.8"} {"level":"info","ts":1556651267.979987,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"} {"level":"info","ts":1556651267.9799957,"logger":"cmd","msg":"Version of operator-sdk: v0.7.0+git"} {"level":"info","ts":1556651267.9800034,"logger":"cmd","msg":"Go Version: go1.10.8"} {"level":"info","ts":1556651267.980006,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"} {"level":"info","ts":1556651267.9800084,"logger":"cmd","msg":"Version of operator-sdk: v0.7.0+git"} {"level":"info","ts":1556651267.9800231,"logger":"cmd","msg":"Watching namespace.","Namespace":"kubevirt-hyperconverged"} {"level":"info","ts":1556651268.1199892,"logger":"leader","msg":"Trying to become the leader."} {"level":"error","ts":1556651268.204345,"logger":"cmd","msg":"Failed to become leader.","Namespace":"kubevirt-hyperconverged","error":"required env POD_NAME not set, please configure downward API","stacktrace":"github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible.Run\n\t/go/src/github.com/operator-framework/operator-sdk/pkg/ansible/run.go:83\ngithub.com/operator-framework/operator-sdk/cmd/operator-sdk/run.newRunAnsibleCmd.func1\n\t/go/src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/run/ansible.go:38\ngithub.com/operator-framework/operator-sdk/vendor/github.com/spf13/cobra.(*Command).execute\n\t/go/src/github.com/operator-framework/operator-sdk/vendor/github.com/spf13/cobra/command.go:762\ngithub.com/operator-framework/operator-sdk/vendor/github.com/spf13/cobra.(*Command).ExecuteC\n\t/go/src/github.com/operator-framework/operator-sdk/vendor/github.com/spf13/cobra/command.go:852\ngithub.com/operator-framework/operator-sdk/vendor/github.com/spf13/cobra.(*Command).Execute\n\t/go/src/github.com/operator-framework/operator-sdk/vendor/github.com/spf13/cobra/command.go:800\nmain.main\n\t/go/src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/main.go:61\nruntime.main\n\t/opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/proc.go:198"} Error: required env POD_NAME not set, please configure downward API Usage: operator-sdk run ansible [flags] Flags: -h, --help help for ansible --inject-owner-ref The ansible operator will inject owner references unless this flag is false (default true) --max-workers int Maximum number of workers to use. Overridden by environment variable. (default 1) --reconcile-period duration Default reconcile period for controllers (default 1m0s) --watches-file string Path to the watches file to use (default "./watches.yaml") --zap-devel Enable zap development mode (changes defaults to console encoder, debug log level, and disables sampling) --zap-encoder encoder Zap log encoding ('json' or 'console') --zap-level level Zap log level (one of 'debug', 'info', 'error' or any integer value > 0) (default info) --zap-sample sample Enable zap log sampling. Sampling will be disabled for integer log levels > 1