Bug 1937097 - Add retries to opm index add
Summary: Add retries to opm index add
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OLM
Version: 4.7
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.7.z
Assignee: Joe Lanford
QA Contact: kuiwang
URL:
Whiteboard:
Depends On: 1951652
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-03-09 19:43 UTC by Luiz Carvalho
Modified: 2021-05-19 15:15 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1951652 (view as bug list)
Environment:
Last Closed: 2021-05-19 15:15:42 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github operator-framework operator-registry pull 633 0 None open Retry failures when pulling images 2021-04-13 21:15:45 UTC
Github operator-framework operator-registry pull 643 0 None open Bug 1937097: Add retries to opm index add 2021-04-22 13:11:01 UTC
Red Hat Product Errata RHBA-2021:1550 0 None None None 2021-05-19 15:15:57 UTC

Description Luiz Carvalho 2021-03-09 19:43:06 UTC
Description of problem:
When opm index add is called with hundreds of bundle images, the request may fail if one of the images cannot be accessed temporarily due to registry hiccups, including registry.redhat.io and registry.access.redhat.com.

When a new minor version of OpenShift is being prepared, a new index image for that version is created. It's common to take most of the existing bundle images from the current version and add them to this new index. Today, the v4.7 index contains 276 bundle images. This is only expected to grow.


Version-Release number of selected component (if applicable):
v1.16.0

How reproducible:
Always when a large number of bundle images are used. Rarely, if only a handful.

Steps to Reproduce:
1. opm index add --bundles [...]

Actual results:
The opm operation fails due to registry errors.

Expected results:
The opm operation retries accessing a bundle image from a registry on registry errors, e.g. 5xx, connection errors, etc.

This will improve opm's resiliency.

Additional info:

Comment 3 Joe Lanford 2021-04-13 21:16:19 UTC
Upstream PR is posted here: https://github.com/operator-framework/operator-registry/pull/633

Comment 6 kuiwang 2021-04-29 06:02:25 UTC
LGTM

--
[root@preserve-olm-env operator-registry]# ./bin/opm version
Version: version.Version{OpmVersion:"v1.15.4-11-gfb7196f", GitCommit:"fb7196f", BuildDate:"2021-04-29T02:22:50Z", GoOs:"linux", GoArch:"amd64"}
[root@preserve-olm-env operator-registry]# cd /data/kuiwang/cases/1937097
[root@preserve-olm-env 1937097]# 
[root@preserve-olm-env 1937097]# oc image extract registry.redhat.io/redhat/redhat-operator-index:v4.7  --file=/database/index.db
W0429 06:48:03.493649   31246 manifest.go:440] Chose linux/amd64 manifest from the manifest list.
[root@preserve-olm-env 1937097]# bundle_list="";x=0;for bundle in $(sqlite3 index.db "select bundlepath from operatorbundle"); do bundle_list="${bundle_list},${bundle}";x=$(($x + 1));if [ $x -gt 50   ] ; then echo $x; break; fi; done
51
[root@preserve-olm-env 1937097]# bundle_list="${bundle_list:1}"
[root@preserve-olm-env 1937097]# 
[root@preserve-olm-env 1937097]# ../../operator-registry/bin/opm index add -b ${bundle_list} -t quay.io/kuiwang/redhat_operator:v1 -p podman
...
INFO[0114] [podman build --format docker -f index.Dockerfile851421109 -t quay.io/kuiwang/redhat_operator:v1 .]  bundles="[registry.redhat.io/3scale-amp2/3scale-rhel7-operator-metadata@sha256:55166d313d270ae4740a260149bfb41df1af8214744e9c38abef07add49c95de registry.redhat.io/3scale-amp2/3scale-rhel7-operator-metadata@sha256:bb7dc249eb517b6e3ffd7d4ceb19884ad4af97f16835bf58e1930534ca7e4764 registry.redhat.io/3scale-amp2/3scale-rhel7-operator-metadata@sha256:523ca71b96847059b68f6def61e4735a52315757395556f37977e083aef305de registry.redhat.io/3scale-amp2/3scale-rhel7-operator-metadata@sha256:9e2588646a095b3067cd785910c7cde820b7b7aaf864d5600a577a13a5bd69c4 registry.redhat.io/3scale-amp2/3scale-rhel7-operator-metadata@sha256:2d1d177e854d5065d872b63502b2ac08bbcb9855ab1f0609106c31ef13f8d67b registry.redhat.io/3scale-amp2/3scale-rhel7-operator-metadata@sha256:3d51773c86faf0a06661d824d25e63bb704fdd45740131ab5c2c3cff2f3afe20 registry.redhat.io/3scale-amp2/3scale-rhel7-operator-metadata@sha256:361ee922e6460100781906421e3b4339850785fe1d1559b17be6555ab7684a52 registry.redhat.io/3scale-amp2/3scale-rhel7-operator-metadata@sha256:a4bd65860deeca86308b783186e34f7f947e5bfe072eee459091d698fac51409 registry.redhat.io/3scale-amp2/3scale-rhel7-operator-metadata@sha256:27a445a5c1950513cf5088dc96e6768c717f3079dfa445775768ea573a1436f7 registry.redhat.io/3scale-amp2/3scale-rhel7-operator-metadata@sha256:65f6dc458d864761dc05f265d2e679104510bb38c5a0e5a31e2268ec674b9ebe registry.redhat.io/3scale-amp2/3scale-rhel7-operator-metadata@sha256:996c4b5c28ba35b640f2ccf12f49a477bdd65e69c3741f0253c850aaf14f0176 registry.redhat.io/3scale-amp2/3scale-rhel7-operator-metadata@sha256:d3f06246be00c7a0e8474eee22791a9b0fcd64863bcd45a19d84e5dc88bc075a registry.redhat.io/3scale-amp2/3scale-rhel7-operator-metadata@sha256:f698cf1102a1847cba810f5be68c264223a37f6424c3e4902465111b5b74d496 registry.redhat.io/rhacm2/acm-operator-bundle@sha256:280bf802843fab435a2946a49324900223f14ae01abbb113624bc056b5ab0188 registry.redhat.io/rhacm2/acm-operator-bundle@sha256:1e1cd6a2950349e3e619e2ffe15cd10d36a5b15e1a9e3b0024bdb11a2355b7ca registry.redhat.io/rhacm2/acm-operator-bundle@sha256:ea42543f1127fd6ec53cf7f6c7f61f3e0b62f1b210844584d89d60c4bf53fef9 registry.redhat.io/rhacm2/acm-operator-bundle@sha256:75c32ced4add3c6c583daf0bcf81b13cbeb99fd2e24f2c3261fb95ac7bcec4f1 registry.redhat.io/rhacm2/acm-operator-bundle@sha256:c068e6f697764fe0aa82f5572450bbf6cc80d9670a7cb65a8753afa9fa9f8bf6 registry.redhat.io/rhacm2/acm-operator-bundle@sha256:1af785b35226525dc1265123bc274bccb025439ea396e560d0a562791f2c8217 registry.redhat.io/rhacm2/acm-operator-bundle@sha256:3a8c3261d61779907a9a86bb4a2699bd8c60f3ec6466cf6656034b7eda127bc6 registry.redhat.io/rhacm2/acm-operator-bundle@sha256:d97ae02a20eaaa1a0770710a7b5808787d9617ce63c03d3c3f08a11f5bf41aa3 registry.redhat.io/rhacm2/acm-operator-bundle@sha256:e77e15271554a64192f4c80a864efe4f082caeeb94d72f10165beb5cba043ebf registry.redhat.io/rhacm2/acm-operator-bundle@sha256:c86057b81e3f4a2979f6d5e2ddcbb9fc31549f8d528497a4924e53b26fd75c8c registry.redhat.io/rhacm2/acm-operator-bundle@sha256:a0a2be3279d03955acf2c2471384c819d45ee765401242eeac20751fe3c835fd registry.redhat.io/rhacm2/acm-operator-bundle@sha256:76435cfe5728bbcacabb1a444ca45df913a7d5a8541b0cc40496cd11d77865db registry.redhat.io/rhacm2/acm-operator-bundle@sha256:f02648187cb8cc1c34a6c463ce7f307cab7016da4c06e836971b9bd23d967394 registry.redhat.io/rhacm2/acm-operator-bundle@sha256:9725b91409219faf9e37bab93529ca21bfd209aab3395403866275a3bd08f072 registry.redhat.io/rhacm2/acm-operator-bundle@sha256:510b81899ac25c4318e31a7a44b5c2645fc14e24cc463847f19c5229e8d87c6f registry.redhat.io/rhacm2/acm-operator-bundle@sha256:5d6c8f7a046eaebd4cee3ad10151bfa3bb176df8e20cd475db66becdcd7356fd registry.redhat.io/rhacm2/acm-operator-bundle@sha256:4e4fc6cdfec03a175ada82a98044ba6ae80e5d7fc096a24bc5f2977f49d6c70b registry.redhat.io/rhacm2/acm-operator-bundle@sha256:55906425c2619315d98a9f2b3b014379bfa14875592d18a550a93344097c5496 registry.redhat.io/rhacm2/acm-operator-bundle@sha256:96e323e923dfb556daadc87e990eb89277c21926e56e378bae60bf1d2a46931a registry.redhat.io/rhacm2/acm-operator-bundle@sha256:adba2e6b585bde1c05d041097a9f19b0753230480032aa44ad1b27fecbf819f5 registry.redhat.io/rhacm2/acm-operator-bundle@sha256:82f8508111f307c25e458a2283e86a21eaeeaa2cd0b335766174f324f6bd392d registry.redhat.io/amq7/amq-broker-operator-bundle@sha256:5a5f53ec96ddb46f6c25bf7319ecb1a86e6754da0d09927c917ecea037eb6d05 registry.redhat.io/amq7/amq-broker-operator-bundle@sha256:1d8aebf3ee175b795bb14c385d1183bd9da5c2fdce4005d09b01ddc2a3b1b0aa registry.redhat.io/amq7/amq-broker-operator-bundle@sha256:86853f5d81be450a999ba261c60bfe1e429a4a8312d7ef5dedb2cd66be118147 registry.redhat.io/amq7/amq-broker-operator-bundle@sha256:93eb77beb3df602b64e59521459f4b389777ad3409e5dc5715e672af3e1059b2 registry.redhat.io/amq7/amq-broker-operator-bundle@sha256:63b9c3d4b7f7b06ce10f57afec75cb4c39470a800d7ab26d7cc5d9b5876ef347 registry.redhat.io/amq7/amq-broker-operator-bundle@sha256:4677056566e09daf0a316e24abdba542c29c2194cd98b6e8af50f654a930a97d registry.redhat.io/amq7/amq-broker-lts-operator-bundle@sha256:154d343f8646c2fe8d35c235e1fa112ece443f08a9c18bf0029892741dd73b9c registry.redhat.io/amq7/amq-broker-lts-operator-bundle@sha256:b480e64de68bbe7c271623cf9a9f88067fa7d1dd4dc54f192ca505ecfb25dd17 registry.redhat.io/amq7/amq-online-1-controller-manager-rhel7-operator-metadata@sha256:ed1658893b5e78ea6d684dac7f237a80eb1a5fe146996c39152c077ea59f0d61 registry.redhat.io/amq7/amq-online-1-controller-manager-rhel7-operator-metadata@sha256:6195b018194b457eb6bad2c53381bdfa4d84329e86d561580a63a9c2faab66d9 registry.redhat.io/amq7/amq-online-1-controller-manager-rhel7-operator-metadata@sha256:28157b46dcee1e5fffef9260aa9639df03cca2d8e84ccd63bf1fd966c17c10d5 registry.redhat.io/amq7/amq-online-1-controller-manager-rhel7-operator-metadata@sha256:f0f899b90e0ab0e9cc9818e4181beccf178543388c14570da0fd8dda0b4abaa9 registry.redhat.io/amq7/amq-online-1-controller-manager-rhel7-operator-metadata@sha256:b350796e748ac56aeaf71711f4eaae06d2b9f77fc1c9aeec1680c9ab096e663c registry.redhat.io/amq7/amq-online-1-controller-manager-rhel7-operator-metadata@sha256:609f34c93dfc53fb853c45d395fcb5c07b6d96e7ab075b036c785203415cfc11 registry.redhat.io/amq7/amq-online-1-controller-manager-rhel7-operator-metadata@sha256:81fabc8e6bdc05a772fbf4401331e8ccc568593dafada38fff4c8aaf0565ba06 registry.redhat.io/amq7/amq-online-1-controller-manager-rhel7-operator-metadata@sha256:e81aac14ba253b4a89370f8aabfebf491f34621cf8d54efd6b8b3796ca8732bb registry.redhat.io/amq7/amq-online-1-controller-manager-rhel7-operator-metadata@sha256:c5b5ba7265c5ac8e77ff9836746a8ac9ca5e1967270adffffdfe32d8a14968cf]"
[root@preserve-olm-env 1937097]# 
[root@preserve-olm-env 1937097]# podman image prune -f -a
cd21a511cfa59c47c59ccee788b2befb0821554e58f36dcba30193ab3f779c86
6b5ba8fe5426888c28a5edbfb7cfb4d8b6dc1794da4d7767ea12420f5f0f15b1
...
9941df07281bd65aae777facd0d9117fae4ccda4801a0d60756166984fdaf450
e89efb07796109de9760060907fd809ffd328075593cb0847f547098a5d6e741
quay.io/kuiwang/redhat_operator:v1
[root@preserve-olm-env 1937097]#
[root@preserve-olm-env 1937097]# ../../operator-registry/bin/opm index add -b ${bundle_list} -t quay.io/kuiwang/redhat_operator:v1
...
INFO[0043] [podman build --format docker -f index.Dockerfile529500738 -t quay.io/kuiwang/redhat_operator:v1 .]  bundles="[registry.redhat.io/3scale-amp2/3scale-rhel7-operator-metadata@sha256:55166d313d270ae4740a260149bfb41df1af8214744e9c38abef07add49c95de registry.redhat.io/3scale-amp2/3scale-rhel7-operator-metadata@sha256:bb7dc249eb517b6e3ffd7d4ceb19884ad4af97f16835bf58e1930534ca7e4764 registry.redhat.io/3scale-amp2/3scale-rhel7-operator-metadata@sha256:523ca71b96847059b68f6def61e4735a52315757395556f37977e083aef305de registry.redhat.io/3scale-amp2/3scale-rhel7-operator-metadata@sha256:9e2588646a095b3067cd785910c7cde820b7b7aaf864d5600a577a13a5bd69c4 registry.redhat.io/3scale-amp2/3scale-rhel7-operator-metadata@sha256:2d1d177e854d5065d872b63502b2ac08bbcb9855ab1f0609106c31ef13f8d67b registry.redhat.io/3scale-amp2/3scale-rhel7-operator-metadata@sha256:3d51773c86faf0a06661d824d25e63bb704fdd45740131ab5c2c3cff2f3afe20 registry.redhat.io/3scale-amp2/3scale-rhel7-operator-metadata@sha256:361ee922e6460100781906421e3b4339850785fe1d1559b17be6555ab7684a52 registry.redhat.io/3scale-amp2/3scale-rhel7-operator-metadata@sha256:a4bd65860deeca86308b783186e34f7f947e5bfe072eee459091d698fac51409 registry.redhat.io/3scale-amp2/3scale-rhel7-operator-metadata@sha256:27a445a5c1950513cf5088dc96e6768c717f3079dfa445775768ea573a1436f7 registry.redhat.io/3scale-amp2/3scale-rhel7-operator-metadata@sha256:65f6dc458d864761dc05f265d2e679104510bb38c5a0e5a31e2268ec674b9ebe registry.redhat.io/3scale-amp2/3scale-rhel7-operator-metadata@sha256:996c4b5c28ba35b640f2ccf12f49a477bdd65e69c3741f0253c850aaf14f0176 registry.redhat.io/3scale-amp2/3scale-rhel7-operator-metadata@sha256:d3f06246be00c7a0e8474eee22791a9b0fcd64863bcd45a19d84e5dc88bc075a registry.redhat.io/3scale-amp2/3scale-rhel7-operator-metadata@sha256:f698cf1102a1847cba810f5be68c264223a37f6424c3e4902465111b5b74d496 registry.redhat.io/rhacm2/acm-operator-bundle@sha256:280bf802843fab435a2946a49324900223f14ae01abbb113624bc056b5ab0188 registry.redhat.io/rhacm2/acm-operator-bundle@sha256:1e1cd6a2950349e3e619e2ffe15cd10d36a5b15e1a9e3b0024bdb11a2355b7ca registry.redhat.io/rhacm2/acm-operator-bundle@sha256:ea42543f1127fd6ec53cf7f6c7f61f3e0b62f1b210844584d89d60c4bf53fef9 registry.redhat.io/rhacm2/acm-operator-bundle@sha256:75c32ced4add3c6c583daf0bcf81b13cbeb99fd2e24f2c3261fb95ac7bcec4f1 registry.redhat.io/rhacm2/acm-operator-bundle@sha256:c068e6f697764fe0aa82f5572450bbf6cc80d9670a7cb65a8753afa9fa9f8bf6 registry.redhat.io/rhacm2/acm-operator-bundle@sha256:1af785b35226525dc1265123bc274bccb025439ea396e560d0a562791f2c8217 registry.redhat.io/rhacm2/acm-operator-bundle@sha256:3a8c3261d61779907a9a86bb4a2699bd8c60f3ec6466cf6656034b7eda127bc6 registry.redhat.io/rhacm2/acm-operator-bundle@sha256:d97ae02a20eaaa1a0770710a7b5808787d9617ce63c03d3c3f08a11f5bf41aa3 registry.redhat.io/rhacm2/acm-operator-bundle@sha256:e77e15271554a64192f4c80a864efe4f082caeeb94d72f10165beb5cba043ebf registry.redhat.io/rhacm2/acm-operator-bundle@sha256:c86057b81e3f4a2979f6d5e2ddcbb9fc31549f8d528497a4924e53b26fd75c8c registry.redhat.io/rhacm2/acm-operator-bundle@sha256:a0a2be3279d03955acf2c2471384c819d45ee765401242eeac20751fe3c835fd registry.redhat.io/rhacm2/acm-operator-bundle@sha256:76435cfe5728bbcacabb1a444ca45df913a7d5a8541b0cc40496cd11d77865db registry.redhat.io/rhacm2/acm-operator-bundle@sha256:f02648187cb8cc1c34a6c463ce7f307cab7016da4c06e836971b9bd23d967394 registry.redhat.io/rhacm2/acm-operator-bundle@sha256:9725b91409219faf9e37bab93529ca21bfd209aab3395403866275a3bd08f072 registry.redhat.io/rhacm2/acm-operator-bundle@sha256:510b81899ac25c4318e31a7a44b5c2645fc14e24cc463847f19c5229e8d87c6f registry.redhat.io/rhacm2/acm-operator-bundle@sha256:5d6c8f7a046eaebd4cee3ad10151bfa3bb176df8e20cd475db66becdcd7356fd registry.redhat.io/rhacm2/acm-operator-bundle@sha256:4e4fc6cdfec03a175ada82a98044ba6ae80e5d7fc096a24bc5f2977f49d6c70b registry.redhat.io/rhacm2/acm-operator-bundle@sha256:55906425c2619315d98a9f2b3b014379bfa14875592d18a550a93344097c5496 registry.redhat.io/rhacm2/acm-operator-bundle@sha256:96e323e923dfb556daadc87e990eb89277c21926e56e378bae60bf1d2a46931a registry.redhat.io/rhacm2/acm-operator-bundle@sha256:adba2e6b585bde1c05d041097a9f19b0753230480032aa44ad1b27fecbf819f5 registry.redhat.io/rhacm2/acm-operator-bundle@sha256:82f8508111f307c25e458a2283e86a21eaeeaa2cd0b335766174f324f6bd392d registry.redhat.io/amq7/amq-broker-operator-bundle@sha256:5a5f53ec96ddb46f6c25bf7319ecb1a86e6754da0d09927c917ecea037eb6d05 registry.redhat.io/amq7/amq-broker-operator-bundle@sha256:1d8aebf3ee175b795bb14c385d1183bd9da5c2fdce4005d09b01ddc2a3b1b0aa registry.redhat.io/amq7/amq-broker-operator-bundle@sha256:86853f5d81be450a999ba261c60bfe1e429a4a8312d7ef5dedb2cd66be118147 registry.redhat.io/amq7/amq-broker-operator-bundle@sha256:93eb77beb3df602b64e59521459f4b389777ad3409e5dc5715e672af3e1059b2 registry.redhat.io/amq7/amq-broker-operator-bundle@sha256:63b9c3d4b7f7b06ce10f57afec75cb4c39470a800d7ab26d7cc5d9b5876ef347 registry.redhat.io/amq7/amq-broker-operator-bundle@sha256:4677056566e09daf0a316e24abdba542c29c2194cd98b6e8af50f654a930a97d registry.redhat.io/amq7/amq-broker-lts-operator-bundle@sha256:154d343f8646c2fe8d35c235e1fa112ece443f08a9c18bf0029892741dd73b9c registry.redhat.io/amq7/amq-broker-lts-operator-bundle@sha256:b480e64de68bbe7c271623cf9a9f88067fa7d1dd4dc54f192ca505ecfb25dd17 registry.redhat.io/amq7/amq-online-1-controller-manager-rhel7-operator-metadata@sha256:ed1658893b5e78ea6d684dac7f237a80eb1a5fe146996c39152c077ea59f0d61 registry.redhat.io/amq7/amq-online-1-controller-manager-rhel7-operator-metadata@sha256:6195b018194b457eb6bad2c53381bdfa4d84329e86d561580a63a9c2faab66d9 registry.redhat.io/amq7/amq-online-1-controller-manager-rhel7-operator-metadata@sha256:28157b46dcee1e5fffef9260aa9639df03cca2d8e84ccd63bf1fd966c17c10d5 registry.redhat.io/amq7/amq-online-1-controller-manager-rhel7-operator-metadata@sha256:f0f899b90e0ab0e9cc9818e4181beccf178543388c14570da0fd8dda0b4abaa9 registry.redhat.io/amq7/amq-online-1-controller-manager-rhel7-operator-metadata@sha256:b350796e748ac56aeaf71711f4eaae06d2b9f77fc1c9aeec1680c9ab096e663c registry.redhat.io/amq7/amq-online-1-controller-manager-rhel7-operator-metadata@sha256:609f34c93dfc53fb853c45d395fcb5c07b6d96e7ab075b036c785203415cfc11 registry.redhat.io/amq7/amq-online-1-controller-manager-rhel7-operator-metadata@sha256:81fabc8e6bdc05a772fbf4401331e8ccc568593dafada38fff4c8aaf0565ba06 registry.redhat.io/amq7/amq-online-1-controller-manager-rhel7-operator-metadata@sha256:e81aac14ba253b4a89370f8aabfebf491f34621cf8d54efd6b8b3796ca8732bb registry.redhat.io/amq7/amq-online-1-controller-manager-rhel7-operator-metadata@sha256:c5b5ba7265c5ac8e77ff9836746a8ac9ca5e1967270adffffdfe32d8a14968cf]"
[root@preserve-olm-env 1937097]# 


--

Comment 8 Siddharth Sharma 2021-05-10 17:58:03 UTC
This bug will be shipped as part of next z-stream release 4.7.11 on May 19th, as 4.7.10 was dropped due to a blocker https://bugzilla.redhat.com/show_bug.cgi?id=1958518.

Comment 12 errata-xmlrpc 2021-05-19 15:15:42 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.7.11 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:1550


Note You need to log in before you can comment on or make changes to this bug.