Bug 1979719
| Summary: | Update Service pods in crashloopbackoff authentication information missing | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Dan Seals <dseals> |
| Component: | OpenShift Update Service | Assignee: | Pratik Mahajan <pmahajan> |
| OpenShift Update Service sub component: | operand | QA Contact: | Yang Yang <yanyang> |
| Status: | CLOSED ERRATA | Docs Contact: | Kathryn Alexander <kalexand> |
| Severity: | medium | ||
| Priority: | medium | CC: | jiajliu, lmohanty, pmahajan, wking |
| Version: | 4.7 | ||
| Target Milestone: | --- | ||
| Target Release: | 4.9.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-10-15 16:20:30 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
"authentication information missing for index" is from dk-registry's get_credentials [1], which we call from our read_credentials [2], which we call when setting up the plugin [3]. Ideally something in that chain would say "I'm going to wait and see if I actually need credentials for this registry, before trying to load them from this auth file". Or maybe we try to proactively load them, but then treat AuthInfoMissing as non-fatal until we get a 403 from the registry? Or something. [1]: https://github.com/camallo/dkregistry-rs/blob/854d0da53bef5dd85b5e901123e85d43af97c74e/src/lib.rs#L57-L74 [2]: https://github.com/openshift/cincinnati/blob/915cdf1440c3d7801a5abe71cecdadb2c5901bff/cincinnati/src/plugins/internal/graph_builder/release_scrape_dockerv2/registry/mod.rs#L192-L199 [3]: https://github.com/openshift/cincinnati/blob/915cdf1440c3d7801a5abe71cecdadb2c5901bff/cincinnati/src/plugins/internal/graph_builder/release_scrape_dockerv2/plugin.rs#L115-L121 The operator unconditionally copies the cluster pull secret into a local secret mounted for Cincinnati [1,2,3]. I think that's appropriate, and that we don't want to teach the operator about guessing what Cincinnati wants here. Moving to the operand sub-component for making Cincinnati more relaxed about accepting secrets that do not contain a cred for the target registry in situations where the target registry allows the required access to unauthenticated clients. [1]: https://github.com/openshift/cincinnati-operator/blob/1a746871611d4ccb07aa7599c9eac0ef21df56d1/controllers/updateservice_controller.go#L89-L103 [2]: https://github.com/openshift/cincinnati-operator/blob/1a746871611d4ccb07aa7599c9eac0ef21df56d1/controllers/names.go#L57-L58 [3]: https://github.com/openshift/cincinnati-operator/blob/1a746871611d4ccb07aa7599c9eac0ef21df56d1/controllers/new.go#L56 Reproducing it by using an unauthenticated and insecure registry to store the graph-data image and ocp release images.
# oc get po
NAME READY STATUS RESTARTS AGE
service-555ddd7c8-95h7q 1/2 CrashLoopBackOff 5 3m58s
updateservice-operator-5b7564fd5d-mb4gj 1/1 Running 0 3m59s
# oc logs pod/service-555ddd7c8-95h7q graph-builder
[2021-07-07T07:07:49Z DEBUG graph_builder] application settings:
AppSettings {
address: ::,
credentials_path: None,
mandatory_client_parameters: {},
manifestref_key: "io.openshift.upgrades.graph.release.manifestref",
path_prefix: "",
pause_secs: 300s,
scrape_timeout_secs: None,
port: 8080,
registry: "quay.io",
repository: "openshift-release-dev/ocp-release",
status_address: ::,
status_port: 9080,
verbosity: Trace,
fetch_concurrency: 16,
metrics_required: {
"graph_upstream_raw_releases",
},
plugin_settings: [
ReleaseScrapeDockerv2Settings {
registry: "jliu-registry.qe.devcluster.openshift.com:5000",
repository: "ocp4/openshift4-release-images",
manifestref_key: "io.openshift.upgrades.graph.release.manifestref",
fetch_concurrency: 16,
username: None,
password: None,
credentials_path: Some(
"/var/lib/cincinnati/registry-credentials/.dockerconfigjson",
),
},
OpenshiftSecondaryMetadataParserSettings {
data_directory: "/var/lib/cincinnati/graph-data",
key_prefix: "io.openshift.upgrades.graph",
default_arch: "amd64",
disallowed_errors: {},
},
EdgeAddRemovePlugin {
key_prefix: "io.openshift.upgrades.graph",
remove_all_edges_value: "*",
remove_consumed_metadata: false,
},
],
tracing_endpoint: None,
}
Error: Reading registry credentials from "/var/lib/cincinnati/registry-credentials/.dockerconfigjson"
Caused by:
authentication information missing for index jliu-registry.qe.devcluster.openshift.com:5000
# oc get updateservices.updateservice.operator.openshift.io service -oyaml
apiVersion: updateservice.operator.openshift.io/v1
kind: UpdateService
metadata:
creationTimestamp: "2021-07-07T06:00:47Z"
generation: 1
managedFields:
- apiVersion: updateservice.operator.openshift.io/v1
fieldsType: FieldsV1
fieldsV1:
f:spec:
.: {}
f:graphDataImage: {}
f:releases: {}
f:replicas: {}
manager: Mozilla
operation: Update
time: "2021-07-07T06:00:47Z"
- apiVersion: updateservice.operator.openshift.io/v1
fieldsType: FieldsV1
fieldsV1:
f:status:
.: {}
f:conditions: {}
f:policyEngineURI: {}
manager: update-service-operator
operation: Update
time: "2021-07-07T06:00:48Z"
name: service
namespace: openshift-update-service
resourceVersion: "127235"
selfLink: /apis/updateservice.operator.openshift.io/v1/namespaces/openshift-update-service/updateservices/service
uid: b5c701b1-5a7b-46c9-92be-b1037f6e78b1
spec:
foo: bar
graphDataImage: jliu-registry.qe.devcluster.openshift.com:5000/openshift/graph-data:latest
releases: jliu-registry.qe.devcluster.openshift.com:5000/ocp4/openshift4-release-images
replicas: 1
status:
conditions:
- lastHeartbeatTime: "2021-07-07T07:23:43Z"
lastTransitionTime: "2021-07-07T07:23:43Z"
reason: Success
status: "True"
type: ReconcileCompleted
- lastHeartbeatTime: "2021-07-07T07:23:43Z"
lastTransitionTime: "2021-07-07T07:23:43Z"
reason: CACertFound
status: "True"
type: RegistryCACertFound
policyEngineURI: https://service-policy-engine-route-openshift-update-service.apps.yangyang0707.qe.devcluster.openshift.com
# oc extract secret/pull-secret -n openshift-config --confirm
.dockerconfigjson
# grep jliu .dockerconfigjson
null
So it's reproduced.
Trying to workaround it by adding a dummy authentication for the registry to pull secret
# cp .dockerconfigjson newconfigjson
# vim newconfigjson
{"auths":{"jliu-registry.qe.devcluster.openshift.com:5000":{"auth":"xxxxxxxxx"},<snip>
# oc set data secret/pull-secret -n openshift-config --from-file=.dockerconfigjson=newconfigjson
# oc get po
NAME READY STATUS RESTARTS AGE
service-665665f998-vwxjv 1/2 Running 0 13m
updateservice-operator-5b7564fd5d-mb4gj 1/1 Running 0 52m
# oc logs pod/service-665665f998-vwxjv graph-builder
[2021-07-07T07:42:51Z DEBUG graph_builder] application settings:
AppSettings {
address: ::,
credentials_path: None,
mandatory_client_parameters: {},
manifestref_key: "io.openshift.upgrades.graph.release.manifestref",
path_prefix: "",
pause_secs: 300s,
scrape_timeout_secs: None,
port: 8080,
registry: "quay.io",
repository: "openshift-release-dev/ocp-release",
status_address: ::,
status_port: 9080,
verbosity: Trace,
fetch_concurrency: 16,
metrics_required: {
"graph_upstream_raw_releases",
},
plugin_settings: [
ReleaseScrapeDockerv2Settings {
registry: "jliu-registry.qe.devcluster.openshift.com:5000",
repository: "ocp4/openshift4-release-images",
manifestref_key: "io.openshift.upgrades.graph.release.manifestref",
fetch_concurrency: 16,
username: None,
password: None,
credentials_path: Some(
"/var/lib/cincinnati/registry-credentials/.dockerconfigjson",
),
},
OpenshiftSecondaryMetadataParserSettings {
data_directory: "/var/lib/cincinnati/graph-data",
key_prefix: "io.openshift.upgrades.graph",
default_arch: "amd64",
disallowed_errors: {},
},
EdgeAddRemovePlugin {
key_prefix: "io.openshift.upgrades.graph",
remove_all_edges_value: "*",
remove_consumed_metadata: false,
},
],
tracing_endpoint: None,
}
[2021-07-07T07:42:51Z DEBUG graph_builder::graph] graph update triggered
[2021-07-07T07:42:51Z TRACE cincinnati::plugins] Running next plugin 'release-scrape-dockerv2'
[2021-07-07T07:42:51Z ERROR graph_builder::graph] failed to fetch all release metadata
[2021-07-07T07:42:51Z ERROR graph_builder::graph] http transport error: error sending request for url (https://jliu-registry.qe.devcluster.openshift.com:5000/v2/): error trying to connect: error:1408F10B:SSL routines:ssl3_get_record:wrong version number:ssl/record/ssl3_record.c:332:
[2021-07-07T07:42:51Z ERROR graph_builder::graph] error sending request for url (https://jliu-registry.qe.devcluster.openshift.com:5000/v2/): error trying to connect: error:1408F10B:SSL routines:ssl3_get_record:wrong version number:ssl/record/ssl3_record.c:332:
[2021-07-07T07:42:51Z ERROR graph_builder::graph] error trying to connect: error:1408F10B:SSL routines:ssl3_get_record:wrong version number:ssl/record/ssl3_record.c:332:
[2021-07-07T07:42:51Z ERROR graph_builder::graph] error:1408F10B:SSL routines:ssl3_get_record:wrong version number:ssl/record/ssl3_record.c:332:
# curl http://jliu-registry.qe.devcluster.openshift.com:5000/v2/ocp4/openshift4-release-images/tags/list
{"name":"ocp4/openshift4-release-images","tags":["4.7.18-x86_64","4.7.16-x86_64"]}
The credential issue disappears but it ran into a separate issue. I'm using an insecure registry but the request was sent to https://<registry>. Does it support an insecure registry?
@yanyang We do not currently support insecure registries. > We do not currently support insecure registries. Then I'm switching to a registry with certificate but without authentication and attempting to workaround it by adding dummy creds to the cluster pull secret. # curl https://jliu-registry.qe.devcluster.openshift.com/v2/_catalog {"repositories":["ocp4/openshift4","ocp4/openshift4-release-images","openshift/graph-data"]} # podman push jliu-registry.qe.devcluster.openshift.com/openshift/graph-data:1.0 Getting image source signatures Copying blob 668db11eda93 done Copying blob 129d65ee3020 done Copying blob d3ada5af5602 done Writing manifest to image destination Storing signatures podman push works locally. # oc get po NAME READY STATUS RESTARTS AGE service-f9586b4d8-7xmhq 1/2 Running 0 27s updateservice-operator-5b7564fd5d-tg9c8 1/1 Running 0 3h5m # oc logs pod/service-f9586b4d8-7xmhq graph-builder [2021-07-08T09:33:41Z DEBUG graph_builder] application settings: AppSettings { address: ::, credentials_path: None, mandatory_client_parameters: {}, manifestref_key: "io.openshift.upgrades.graph.release.manifestref", path_prefix: "", pause_secs: 300s, scrape_timeout_secs: None, port: 8080, registry: "quay.io", repository: "openshift-release-dev/ocp-release", status_address: ::, status_port: 9080, verbosity: Trace, fetch_concurrency: 16, metrics_required: { "graph_upstream_raw_releases", }, plugin_settings: [ ReleaseScrapeDockerv2Settings { registry: "jliu-registry.qe.devcluster.openshift.com", repository: "ocp4/openshift4-release-images", manifestref_key: "io.openshift.upgrades.graph.release.manifestref", fetch_concurrency: 16, username: None, password: None, credentials_path: Some( "/var/lib/cincinnati/registry-credentials/.dockerconfigjson", ), }, OpenshiftSecondaryMetadataParserSettings { data_directory: "/var/lib/cincinnati/graph-data", key_prefix: "io.openshift.upgrades.graph", default_arch: "amd64", disallowed_errors: {}, }, EdgeAddRemovePlugin { key_prefix: "io.openshift.upgrades.graph", remove_all_edges_value: "*", remove_consumed_metadata: false, }, ], tracing_endpoint: None, } [2021-07-08T09:33:41Z DEBUG graph_builder::graph] graph update triggered [2021-07-08T09:33:41Z TRACE cincinnati::plugins] Running next plugin 'release-scrape-dockerv2' [2021-07-08T09:33:41Z ERROR graph_builder::graph] failed to fetch all release metadata [2021-07-08T09:33:41Z ERROR graph_builder::graph] missing authentication header WWW-Authenticate The registry does not require authentication. It's weird that it fails when the client accesses the registry w/o auth. Updating the test result with osus-v4.9.0-1, the graph-builder failed to create as certificate verify failed.
The certificate is generated by:
$ openssl req -newkey rsa:4096 -nodes -sha256 -keyout osus.key -x509 -days 365 -out osus.crt
And it was added to configmap:
# oc get cm trusted-ca -n openshift-config -oyaml
apiVersion: v1
data:
updateservice-registry: |
-----BEGIN CERTIFICATE-----
<snip>
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
<snip>
-----END CERTIFICATE-----
upshift.mirror-registry.qe.devcluster.openshift.com..5000: |
-----BEGIN CERTIFICATE-----
<snip>
-----END CERTIFICATE-----
yy-registry.qe.devcluster.openshift.com: |
-----BEGIN CERTIFICATE-----
<snip>
-----END CERTIFICATE-----
kind: ConfigMap
metadata:
creationTimestamp: "2021-09-28T02:55:09Z"
name: trusted-ca
namespace: openshift-config
resourceVersion: "244932"
uid: 2ddc6e27-8bd1-41de-986e-fb00a2c18e8d
# oc get po
NAME READY STATUS RESTARTS AGE
service-7b5bbbb584-qtf7g 1/2 Running 0 16m
service-7b5bbbb584-z5vgr 1/2 Running 0 16m
updateservice-operator-7c5d5fb778-slfxr 1/1 Running 0 9h
# oc logs pod/service-7b5bbbb584-qtf7g graph-builder
[2021-09-28T11:56:58Z DEBUG graph_builder] application settings:
AppSettings {
address: ::,
credentials_path: None,
mandatory_client_parameters: {},
manifestref_key: "io.openshift.upgrades.graph.release.manifestref",
path_prefix: "",
pause_secs: 300s,
scrape_timeout_secs: None,
port: 8080,
registry: "quay.io",
repository: "openshift-release-dev/ocp-release",
status_address: ::,
status_port: 9080,
verbosity: Trace,
fetch_concurrency: 16,
metrics_required: {
"graph_upstream_raw_releases",
},
plugin_settings: [
ReleaseScrapeDockerv2Settings {
registry: "yy-registry.qe.devcluster.openshift.com",
repository: "ocp4/openshift4-release-images",
manifestref_key: "io.openshift.upgrades.graph.release.manifestref",
fetch_concurrency: 16,
username: None,
password: None,
credentials_path: Some(
"/var/lib/cincinnati/registry-credentials/.dockerconfigjson",
),
},
OpenshiftSecondaryMetadataParserSettings {
data_directory: "/var/lib/cincinnati/graph-data",
key_prefix: "io.openshift.upgrades.graph",
default_arch: "amd64",
disallowed_errors: {},
},
EdgeAddRemovePlugin {
key_prefix: "io.openshift.upgrades.graph",
remove_all_edges_value: "*",
remove_consumed_metadata: false,
},
],
tracing_endpoint: None,
}
[2021-09-28T11:56:58Z WARN cincinnati::plugins::internal::graph_builder::release_scrape_dockerv2::plugin] Error reading registry credentials from "/var/lib/cincinnati/registry-credentials/.dockerconfigjson". Access to "yy-registry.qe.devcluster.openshift.com" will be unauthenticated: authentication information missing for index yy-registry.qe.devcluster.openshift.com
[2021-09-28T11:56:58Z DEBUG graph_builder::graph] graph update triggered
[2021-09-28T11:56:58Z TRACE cincinnati::plugins] Running next plugin 'release-scrape-dockerv2'
[2021-09-28T11:56:59Z ERROR graph_builder::graph] failed to fetch all release metadata
[2021-09-28T11:56:59Z ERROR graph_builder::graph] http transport error: error sending request for url (https://yy-registry.qe.devcluster.openshift.com/v2/): error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915: (self signed certificate)
[2021-09-28T11:56:59Z ERROR graph_builder::graph] error sending request for url (https://yy-registry.qe.devcluster.openshift.com/v2/): error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915: (self signed certificate)
[2021-09-28T11:56:59Z ERROR graph_builder::graph] error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915: (self signed certificate)
[2021-09-28T11:56:59Z ERROR graph_builder::graph] error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915: (self signed certificate)
[2021-09-28T11:56:59Z ERROR graph_builder::graph] error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915:
Pratik, could you please help spot the issue? Thanks!
Updating the configmap for CA as below. The graph-builder is created successfully and we can see "Access to xxx will be unauthenticated" displayed in the log.
# oc get cm trusted-ca -n openshift-config -oyaml
apiVersion: v1
data:
updateservice-registry: |
-----BEGIN CERTIFICATE-----
<cert of yy-registry>
-----END CERTIFICATE-----
yy-registry.qe.devcluster.openshift.com: |
-----BEGIN CERTIFICATE-----
<cert of yy-registry>
-----END CERTIFICATE-----
kind: ConfigMap
metadata:
creationTimestamp: "2021-09-28T02:55:09Z"
name: trusted-ca
namespace: openshift-config
resourceVersion: "628434"
uid: 2ddc6e27-8bd1-41de-986e-fb00a2c18e8d
# oc get pod
NAME READY STATUS RESTARTS AGE
service-7b5bbbb584-kq5ll 2/2 Running 0 12s
updateservice-operator-7c5d5fb778-slfxr 1/1 Running 0 26h
# oc logs pod/service-7b5bbbb584-kq5ll graph-builder
[2021-09-29T05:42:49Z WARN cincinnati::plugins::internal::graph_builder::release_scrape_dockerv2::plugin] Error reading registry credentials from "/var/lib/cincinnati/registry-credentials/.dockerconfigjson". Access to "yy-registry.qe.devcluster.openshift.com" will be unauthenticated: authentication information missing for index yy-registry.qe.devcluster.openshift.com
Moving it to verified state.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (RHEA: OSUS enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:3735 |
OCP 4.7 Update Service pods in CLBO $ oc --kubeconfig kubeconfig get pods -n openshift-update-service NAME READY STATUS RESTARTS AGE updateservice-76d5897f87-dpgdn 1/2 CrashLoopBackOff 7 13m updateservice-76d5897f87-vxjqk 1/2 CrashLoopBackOff 7 13m updateservice-operator-575c8fd85b-c5xv4 1/1 Running 0 126m $ oc --kubeconfig kubeconfig -n openshift-update-service logs updateservice-76d5897f87-vxjqk graph-builder ..... Error: Reading registry credentials from "/var/lib/cincinnati/registry-credentials/.dockerconfigjson" Caused by: authentication information missing for index docker.odyssey.apps.<domain> The registry docker.odyssey.apps.<domain> does not require credentials. No credentials have been entered in the /var/lib/cincinnati/registry-credentials/.dockerconfigjson other than the pull secret. Appears that if no credentials are provided in the .dockerconfigjson file the pods will not start.