Bug 1826443 - CSV stuck in installing phase
Summary: CSV stuck in installing phase
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OLM
Version: 4.4
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.5.0
Assignee: Alexander Greene
QA Contact: kuiwang
URL:
Whiteboard:
Depends On:
Blocks: 1827000
TreeView+ depends on / blocked
 
Reported: 2020-04-21 17:26 UTC by Alexander Greene
Modified: 2020-07-13 17:30 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: OLM allows users to specify volumes and volumeMounts via the subscriptionConfig field within a Subscription. Using this feature updates the deployment defined in the CSV. Occasionally, OLM would not have the subscription created for a CSV in its cache, and the CSV would be placed in the "installing phase" without creating the deployment with the volumes / volumeMounts defined in the Subscription. OLM would then be unable to move the CSV into the "Succeeded phase" because the calculated deployment hash would not equal the actual deployment hash on the deployment. This error would not be resolved because OLM does not update/recreate the deployment in the "installing phase", and the issue would persist until 5 minutes passed when OLM would re-sync CSVs. Consequence: OLM would occasionally be delayed while installing CSVs. Fix: If OLM encounters a deployment hash error when installing a CSV, OLM will recreate the deployment. Result: OLM is no longer delayed by an incorrect deployment hash.
Clone Of:
Environment:
Last Closed: 2020-07-13 17:29:56 UTC
Target Upstream Version:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github operator-framework operator-lifecycle-manager pull 1472 0 None closed Bug 1826443: Pod Config Deployment Hash Error 2020-10-30 06:11:15 UTC
Red Hat Product Errata RHBA-2020:2409 0 None None None 2020-07-13 17:30:10 UTC

Description Alexander Greene 2020-04-21 17:26:24 UTC 
Description of problem:
If OLM doesn't have the subscription in its cache when it initially creates a deployment, the CSV will hang on "Installing" until it reaches the five-minute timeout, then succeed on a retry. OLM should be updated to roll out the deployment upon noticing the issue.

Version-Release number of selected component (if applicable):
4.4

How reproducible:
Frequently - not always.

Steps to Reproduce:
The easiest way to reproduce this issue is to run the TestCreateNewSubscriptionWithPodConfig e2e-test.

Actual results:
The CSV is stuck in the `Installing` phase for 5 minutes.

Expected results:
The CSV is reaches the `Succeeded` phase quickly.


Additional info:
Comment 3 Ben Luddy 2020-05-07 20:24:31 UTC 
Please remember to remove the test band-aid (located here https://github.com/operator-framework/operator-lifecycle-manager/blob/master/test/e2e/subscription_e2e_test.go#L1215-L1218) as part of the resolution for this issue.
Comment 8 kuiwang 2020-05-28 13:36:20 UTC 
verify it on 4.5. LGTM

--
kuiwang@Kuis-MacBook-Pro operator-lifecycle-manager % git remote -v    
origin  git:operator-framework/operator-lifecycle-manager.git (fetch)
origin  git:operator-framework/operator-lifecycle-manager.git (push)
kuiwang@Kuis-MacBook-Pro operator-lifecycle-manager % git pull origin master
From github.com:operator-framework/operator-lifecycle-manager
 * branch              master     -> FETCH_HEAD
Already up to date.
kuiwang@Kuis-MacBook-Pro operator-lifecycle-manager % git diff
diff --git a/test/e2e/subscription_e2e_test.go b/test/e2e/subscription_e2e_test.go
index 041d4f80..6d67875a 100644
--- a/test/e2e/subscription_e2e_test.go
+++ b/test/e2e/subscription_e2e_test.go
@@ -1062,7 +1062,7 @@ var _ = Describe("Subscription", func() {
                }
        })
 
-       It("creation with pod config", func() {
+       FIt("creation with pod config", func() {
 
                defer cleaner.NotifyTestComplete(true)
 
run this case "creation with pod config" for 10 times. no fail for this case
kuiwang@Kuis-MacBook-Pro operator-lifecycle-manager % for i in `seq 10`; do go test -mod=vendor -count=1 -failfast -tags=local -covermode=count -coverpkg ./pkg/controller/...  -test.v -test.timeout 30m  ./test/e2e/... -kubeconfig=${KUBECONFIG:-~/.kube/config} -namespace=openshift-operator-lifecycle-manager -olmNamespace=openshift-operator-lifecycle-manager; sleep 10 ;done
=== RUN   TestEndToEnd
Running Suite: End-to-end
=========================
Random Seed: 1590646559
Will run 1 of 110 specs
 
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSwaiting for catalog pod podconfig-catsrc-cj4dh to be available (for sync)
catalog podconfig-catsrc-cj4dh pod with address podconfig-catsrc-cj4dh.openshift-operator-lifecycle-manager.svc:50051
 
------------------------------
• [SLOW TEST:43.651 seconds]
Subscription
/Users/kuiwang/GoProject/go-origin/src/github.com/operator-framework/operator-lifecycle-manager/test/e2e/subscription_e2e_test.go:39
  creation with pod config
  /Users/kuiwang/GoProject/go-origin/src/github.com/operator-framework/operator-lifecycle-manager/test/e2e/subscription_e2e_test.go:1065
------------------------------
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
Ran 1 of 110 Specs in 44.623 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 109 Skipped
PASS | FOCUSED
FAIL    github.com/operator-framework/operator-lifecycle-manager/test/e2e       46.220s
?       github.com/operator-framework/operator-lifecycle-manager/test/e2e/assets/chart  [no test files]
?       github.com/operator-framework/operator-lifecycle-manager/test/e2e/ctx   [no test files]
?       github.com/operator-framework/operator-lifecycle-manager/test/e2e/data/kiali-image-registry     [no test files]
?       github.com/operator-framework/operator-lifecycle-manager/test/e2e/dsl   [no test files]
?       github.com/operator-framework/operator-lifecycle-manager/test/e2e/wait  [no test files]
FAIL
=== RUN   TestEndToEnd
Running Suite: End-to-end
=========================
Random Seed: 1590646617
Will run 1 of 110 specs
 
SSSSSSSSSSSSSSSSSSSSSSwaiting for catalog pod podconfig-catsrc-n778w to be available (for sync)
waiting for catalog pod podconfig-catsrc-n778w to be available (for sync)
catalog podconfig-catsrc-n778w pod with address podconfig-catsrc-n778w.openshift-operator-lifecycle-manager.svc:50051
 
------------------------------
• [SLOW TEST:41.910 seconds]
Subscription
/Users/kuiwang/GoProject/go-origin/src/github.com/operator-framework/operator-lifecycle-manager/test/e2e/subscription_e2e_test.go:39
  creation with pod config
  /Users/kuiwang/GoProject/go-origin/src/github.com/operator-framework/operator-lifecycle-manager/test/e2e/subscription_e2e_test.go:1065
------------------------------
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
Ran 1 of 110 Specs in 42.661 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 109 Skipped
PASS | FOCUSED
FAIL    github.com/operator-framework/operator-lifecycle-manager/test/e2e       43.652s
?       github.com/operator-framework/operator-lifecycle-manager/test/e2e/assets/chart  [no test files]
?       github.com/operator-framework/operator-lifecycle-manager/test/e2e/ctx   [no test files]
?       github.com/operator-framework/operator-lifecycle-manager/test/e2e/data/kiali-image-registry     [no test files]
?       github.com/operator-framework/operator-lifecycle-manager/test/e2e/dsl   [no test files]
?       github.com/operator-framework/operator-lifecycle-manager/test/e2e/wait  [no test files]
FAIL
=== RUN   TestEndToEnd
Running Suite: End-to-end
=========================
Random Seed: 1590646674
Will run 1 of 110 specs
 
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSwaiting for catalog pod podconfig-catsrc-6mmqv to be available (for sync)
waiting for catalog pod podconfig-catsrc-6mmqv to be available (for sync)
waiting for catalog pod podconfig-catsrc-6mmqv to be available (for sync)
waiting for catalog pod podconfig-catsrc-6mmqv to be available (for sync)
catalog podconfig-catsrc-6mmqv pod with address podconfig-catsrc-6mmqv.openshift-operator-lifecycle-manager.svc:50051
 
------------------------------
• [SLOW TEST:36.210 seconds]
Subscription
/Users/kuiwang/GoProject/go-origin/src/github.com/operator-framework/operator-lifecycle-manager/test/e2e/subscription_e2e_test.go:39
  creation with pod config
  /Users/kuiwang/GoProject/go-origin/src/github.com/operator-framework/operator-lifecycle-manager/test/e2e/subscription_e2e_test.go:1065
------------------------------
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
Ran 1 of 110 Specs in 36.991 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 109 Skipped
PASS | FOCUSED
FAIL    github.com/operator-framework/operator-lifecycle-manager/test/e2e       37.970s
?       github.com/operator-framework/operator-lifecycle-manager/test/e2e/assets/chart  [no test files]
?       github.com/operator-framework/operator-lifecycle-manager/test/e2e/ctx   [no test files]
?       github.com/operator-framework/operator-lifecycle-manager/test/e2e/data/kiali-image-registry     [no test files]
?       github.com/operator-framework/operator-lifecycle-manager/test/e2e/dsl   [no test files]
?       github.com/operator-framework/operator-lifecycle-manager/test/e2e/wait  [no test files]
FAIL
=== RUN   TestEndToEnd
Running Suite: End-to-end
=========================
Random Seed: 1590646725
Will run 1 of 110 specs
 
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSwaiting for catalog pod podconfig-catsrc-wlmnz to be available (for sync)
catalog podconfig-catsrc-wlmnz pod with address podconfig-catsrc-wlmnz.openshift-operator-lifecycle-manager.svc:50051
 
------------------------------
• [SLOW TEST:42.275 seconds]
Subscription
/Users/kuiwang/GoProject/go-origin/src/github.com/operator-framework/operator-lifecycle-manager/test/e2e/subscription_e2e_test.go:39
  creation with pod config
  /Users/kuiwang/GoProject/go-origin/src/github.com/operator-framework/operator-lifecycle-manager/test/e2e/subscription_e2e_test.go:1065
------------------------------
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
Ran 1 of 110 Specs in 42.960 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 109 Skipped
PASS | FOCUSED
FAIL    github.com/operator-framework/operator-lifecycle-manager/test/e2e       43.857s
?       github.com/operator-framework/operator-lifecycle-manager/test/e2e/assets/chart  [no test files]
?       github.com/operator-framework/operator-lifecycle-manager/test/e2e/ctx   [no test files]
?       github.com/operator-framework/operator-lifecycle-manager/test/e2e/data/kiali-image-registry     [no test files]
?       github.com/operator-framework/operator-lifecycle-manager/test/e2e/dsl   [no test files]
?       github.com/operator-framework/operator-lifecycle-manager/test/e2e/wait  [no test files]
FAIL
=== RUN   TestEndToEnd
Running Suite: End-to-end
=========================
Random Seed: 1590646781
Will run 1 of 110 specs
 
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSwaiting for catalog pod podconfig-catsrc-rzxh2 to be available (for sync)
waiting for catalog pod podconfig-catsrc-rzxh2 to be available (for sync)
waiting for catalog pod podconfig-catsrc-rzxh2 to be available (for sync)
catalog podconfig-catsrc-rzxh2 pod with address podconfig-catsrc-rzxh2.openshift-operator-lifecycle-manager.svc:50051
 
------------------------------
• [SLOW TEST:40.420 seconds]
Subscription
/Users/kuiwang/GoProject/go-origin/src/github.com/operator-framework/operator-lifecycle-manager/test/e2e/subscription_e2e_test.go:39
  creation with pod config
  /Users/kuiwang/GoProject/go-origin/src/github.com/operator-framework/operator-lifecycle-manager/test/e2e/subscription_e2e_test.go:1065
------------------------------
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
Ran 1 of 110 Specs in 41.291 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 109 Skipped
PASS | FOCUSED
FAIL    github.com/operator-framework/operator-lifecycle-manager/test/e2e       42.263s
?       github.com/operator-framework/operator-lifecycle-manager/test/e2e/assets/chart  [no test files]
?       github.com/operator-framework/operator-lifecycle-manager/test/e2e/ctx   [no test files]
?       github.com/operator-framework/operator-lifecycle-manager/test/e2e/data/kiali-image-registry     [no test files]
?       github.com/operator-framework/operator-lifecycle-manager/test/e2e/dsl   [no test files]
?       github.com/operator-framework/operator-lifecycle-manager/test/e2e/wait  [no test files]
FAIL
=== RUN   TestEndToEnd
Running Suite: End-to-end
=========================
Random Seed: 1590646837
Will run 1 of 110 specs
 
SSSSSSSSSSSSSSSwaiting for catalog pod podconfig-catsrc-htfmw to be available (for sync)
waiting for catalog pod podconfig-catsrc-htfmw to be available (for sync)
catalog podconfig-catsrc-htfmw pod with address podconfig-catsrc-htfmw.openshift-operator-lifecycle-manager.svc:50051
 
------------------------------
• [SLOW TEST:31.018 seconds]
Subscription
/Users/kuiwang/GoProject/go-origin/src/github.com/operator-framework/operator-lifecycle-manager/test/e2e/subscription_e2e_test.go:39
  creation with pod config
  /Users/kuiwang/GoProject/go-origin/src/github.com/operator-framework/operator-lifecycle-manager/test/e2e/subscription_e2e_test.go:1065
------------------------------
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
Ran 1 of 110 Specs in 31.956 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 109 Skipped
PASS | FOCUSED
FAIL    github.com/operator-framework/operator-lifecycle-manager/test/e2e       33.346s
?       github.com/operator-framework/operator-lifecycle-manager/test/e2e/assets/chart  [no test files]
?       github.com/operator-framework/operator-lifecycle-manager/test/e2e/ctx   [no test files]
?       github.com/operator-framework/operator-lifecycle-manager/test/e2e/data/kiali-image-registry     [no test files]
?       github.com/operator-framework/operator-lifecycle-manager/test/e2e/dsl   [no test files]
?       github.com/operator-framework/operator-lifecycle-manager/test/e2e/wait  [no test files]
FAIL
=== RUN   TestEndToEnd
Running Suite: End-to-end
=========================
Random Seed: 1590646883
Will run 1 of 110 specs
 
SSSSSSSSSSSSSwaiting for catalog pod podconfig-catsrc-dtj9k to be available (for sync)
waiting for catalog pod podconfig-catsrc-dtj9k to be available (for sync)
catalog podconfig-catsrc-dtj9k pod with address podconfig-catsrc-dtj9k.openshift-operator-lifecycle-manager.svc:50051
 
------------------------------
• [SLOW TEST:45.937 seconds]
Subscription
/Users/kuiwang/GoProject/go-origin/src/github.com/operator-framework/operator-lifecycle-manager/test/e2e/subscription_e2e_test.go:39
  creation with pod config
  /Users/kuiwang/GoProject/go-origin/src/github.com/operator-framework/operator-lifecycle-manager/test/e2e/subscription_e2e_test.go:1065
------------------------------
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
Ran 1 of 110 Specs in 47.801 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 109 Skipped
PASS | FOCUSED
FAIL    github.com/operator-framework/operator-lifecycle-manager/test/e2e       49.398s
?       github.com/operator-framework/operator-lifecycle-manager/test/e2e/assets/chart  [no test files]
?       github.com/operator-framework/operator-lifecycle-manager/test/e2e/ctx   [no test files]
?       github.com/operator-framework/operator-lifecycle-manager/test/e2e/data/kiali-image-registry     [no test files]
?       github.com/operator-framework/operator-lifecycle-manager/test/e2e/dsl   [no test files]
?       github.com/operator-framework/operator-lifecycle-manager/test/e2e/wait  [no test files]
FAIL
=== RUN   TestEndToEnd
Running Suite: End-to-end
=========================
Random Seed: 1590646945
Will run 1 of 110 specs
 
SSSSSSSSSSSSSwaiting for catalog pod podconfig-catsrc-648f7 to be available (for sync)
catalog podconfig-catsrc-648f7 pod with address podconfig-catsrc-648f7.openshift-operator-lifecycle-manager.svc:50051
 
------------------------------
• [SLOW TEST:38.539 seconds]
Subscription
/Users/kuiwang/GoProject/go-origin/src/github.com/operator-framework/operator-lifecycle-manager/test/e2e/subscription_e2e_test.go:39
  creation with pod config
  /Users/kuiwang/GoProject/go-origin/src/github.com/operator-framework/operator-lifecycle-manager/test/e2e/subscription_e2e_test.go:1065
------------------------------
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
Ran 1 of 110 Specs in 39.499 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 109 Skipped
PASS | FOCUSED
FAIL    github.com/operator-framework/operator-lifecycle-manager/test/e2e       40.664s
?       github.com/operator-framework/operator-lifecycle-manager/test/e2e/assets/chart  [no test files]
?       github.com/operator-framework/operator-lifecycle-manager/test/e2e/ctx   [no test files]
?       github.com/operator-framework/operator-lifecycle-manager/test/e2e/data/kiali-image-registry     [no test files]
?       github.com/operator-framework/operator-lifecycle-manager/test/e2e/dsl   [no test files]
?       github.com/operator-framework/operator-lifecycle-manager/test/e2e/wait  [no test files]
FAIL
=== RUN   TestEndToEnd
Running Suite: End-to-end
=========================
Random Seed: 1590646998
Will run 1 of 110 specs
 
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSwaiting for catalog pod podconfig-catsrc-snnpz to be available (for sync)
waiting for catalog pod podconfig-catsrc-snnpz to be available (for sync)
waiting for catalog pod podconfig-catsrc-snnpz to be available (for sync)
catalog podconfig-catsrc-snnpz pod with address podconfig-catsrc-snnpz.openshift-operator-lifecycle-manager.svc:50051
 
------------------------------
• [SLOW TEST:47.418 seconds]
Subscription
/Users/kuiwang/GoProject/go-origin/src/github.com/operator-framework/operator-lifecycle-manager/test/e2e/subscription_e2e_test.go:39
  creation with pod config
  /Users/kuiwang/GoProject/go-origin/src/github.com/operator-framework/operator-lifecycle-manager/test/e2e/subscription_e2e_test.go:1065
------------------------------
Comment 9 errata-xmlrpc 2020-07-13 17:29:56 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409
Note You need to log in before you can comment on or make changes to this bug.