Bug 2242366
| Summary: | [RHOSP 13 to 16 FFU] nova_hybrid_state fails if iscsid container image manifest is v1+json in undercloud registry. | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Matt Flusche <mflusche> |
| Component: | openstack-tripleo-common | Assignee: | Nobody <nobody> |
| Status: | CLOSED DUPLICATE | QA Contact: | David Rosenfeld <drosenfe> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 16.2 (Train) | CC: | dhill, jslagle, lbezdick, ltamagno, mburns, slinaber |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-10-13 13:13:21 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
In the working lab, all v2+json manifests:
# find /var/lib/image-serve/v2/rhosp-rhel8/ -name '*type-map' -exec grep Content-Type {} \;|wc -l
80
# find /var/lib/image-serve/v2/rhosp-rhel8/ -name '*type-map' -exec grep Content-Type {} \;|sort -u
Content-Type: application/vnd.docker.distribution.manifest.v2+json
And a mix in the non-working lab:
# find /var/lib/image-serve/v2/rhosp-rhel8/ -name '*type-map' -exec grep Content-Type {} \;|sort -u
Content-Type: application/vnd.docker.distribution.manifest.v1+json
Content-Type: application/vnd.docker.distribution.manifest.v2+json
# find /var/lib/image-serve/v2/rhosp-rhel8/ -name '*type-map' -exec grep Content-Type {} \;|grep 'v1+json'|wc -l
41
# find /var/lib/image-serve/v2/rhosp-rhel8/ -name '*type-map' -exec grep Content-Type {} \;|grep 'v2+json'|wc -l
12
Both are using the standard registry.redhat.io/rhosp-rhel8 namespace (no satellite).
Maybe we just need this patch
~~~
[dhill@knox tripleo-common]$ git diff stable/train
diff --git a/tripleo_common/image/image_uploader.py b/tripleo_common/image/image_uploader.py
index 505189df..9474c450 100644
--- a/tripleo_common/image/image_uploader.py
+++ b/tripleo_common/image/image_uploader.py
@@ -918,7 +918,7 @@ class BaseImageUploader(object):
)
# prefer docker manifest over oci
manifest_headers = {'Accept': ", ".join([
- MEDIA_MANIFEST_V2 + ";q=1", MEDIA_OCI_MANIFEST_V1 + ";q=0.5"])}
+ MEDIA_MANIFEST_V2 + ";q=1.0", MEDIA_OCI_MANIFEST_V1 + ";q=0.5"])}
try:
manifest_r = RegistrySessionHelper.get(
@@ -1823,7 +1823,7 @@ class PythonImageUploader(BaseImageUploader):
else:
# prefer docker manifest over oci
manifest_headers = {'Accept': ", ".join([
- MEDIA_MANIFEST_V2 + ";q=1", MEDIA_OCI_MANIFEST_V1 + ";q=0.5"])}
+ MEDIA_MANIFEST_V2 + ";q=1.0", MEDIA_OCI_MANIFEST_V1 + ";q=0.5"])}
try:
r = RegistrySessionHelper.get(
session,
~~~
https://review.opendev.org/c/openstack/tripleo-common/+/897716
And for docker, maybe we'll have to backport / fix it given many customers are upgrading from rhosp13 to 16.2 , if we updated quay/registry.redhat.io and it broke older docker , we'll have to fix it down to RHEL 7.9 ... Lukas, could you take a look at this one? This is a 13->16 FFU, after the UC has been upgraded to 16. Is it the same issue as https://bugzilla.redhat.com/show_bug.cgi?id=2240185? Wow this is interesting iteration of the issue. I think we need the async fix that was merged into 16.2 applied here and you will need to refetch the containers. OSP13 does not have the priority patch, it's in OSP16.2 and it's fixed in BZ#2240185 I suggest applying https://review.opendev.org/c/openstack/tripleo-common/+/896319 and removing 16 images from UC and all the overcloud nodes before rerunning image prepare and upload. "Ugly" fix that works on podman but not tested against docker: for manifest_file in `find /var/lib/containers/storage/overlay-images/ -name 'manifest'` do cat <<< $( jq 'if .schemaVersion == 1 then if has("signatures") then . else .signatures=[] end else . end' $manifest_file ) > $manifest_file done *** This bug has been marked as a duplicate of bug 2240185 *** To rephrase my self - images they have in /var/lib/image-serve on Undercloud as broken now so they need to apply patch, remove OSP16 images on UC /var/lib/image-serve and redownload them while also remove all the OSP16 images that got pulled in on the nodes that are on RHEL7.9. |
Description of problem: It seems that current docker version on rhel7.9 (OSP 13) cannot pull containers with manifest type of v1+json from osp 16 undercloud registry (image-serve). If during the container_image_prepare step images are uploaded to director with v1 content type, the containers needed during nova_hybrid_state fail to be pulled by OSP 13 compute nodes. The error is the following: error parsing HTTP 406 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>406 Not Acceptable</title>\n</head><body>\n<h1>Not Acceptable</h1>\n<p>An appropriate representation of the requested resource could not be found on this server.</p>\nAvailable variants:\n<ul>\n<li><a href=\"sha256:dfded702919a2fcbaacd66e86a33ac3da643358b73f0e2524c175ac85bef8a6e%2findex.json\">sha256:dfded702919a2fcbaacd66e86a33ac3da643358b73f0e2524c175ac85bef8a6e/index.json</a> , type application/vnd.docker.distribution.manifest.v1+json</li>\n</ul>\n</body></html>\n" I have two labs, one works and one doesn't; both 16.2.5 current. In the working lab: # grep Content-Type /var/lib/image-serve/v2/rhosp-rhel8/openstack-iscsid/manifests/16.2.type-map Content-Type: application/vnd.docker.distribution.manifest.v2+json In the non-working lab: # grep Content-Type /var/lib/image-serve/v2/rhosp-rhel8/openstack-iscsid/manifests/16.2.5.type-map Content-Type: application/vnd.docker.distribution.manifest.v1+json From a tcpdump[1] we see that the docker client does not accept manifest.v1+json type Another thing to note, in the working lab all container images are type v2+json and the non-working lab has a mix of v1+json and v2+json. docker (osp 13) clients can successfully pull v2+json containers. Both environments are running the same tripleo-common RPMs: openstack-tripleo-common-11.7.1-2.20230308005004.99d0f05.el8ost.noarch python3-tripleo-common-11.7.1-2.20230308005004.99d0f05.el8ost.noarch What causes v1 vs v2 containers to be pulled/uploaded to director local container repo? [1] - decoded http session of docker pull from tcpdump. GET /v2/rhosp-rhel8/openstack-iscsid/manifests/16.2.5 HTTP/1.1 Host: undercloud16.ctlplane.local:8787 User-Agent: docker/1.13.1 go/go1.10.3 kernel/3.10.0-1160.59.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \(linux\)) Accept: application/vnd.docker.distribution.manifest.v2+json Accept: application/vnd.docker.distribution.manifest.list.v2+json Accept: application/vnd.docker.distribution.manifest.v1+prettyjws Accept: application/json Accept-Encoding: gzip Connection: close HTTP/1.1 406 Not Acceptable Date: Thu, 05 Oct 2023 17:22:51 GMT Server: Apache/2.4.37 (Red Hat Enterprise Linux) Alternates: {"sha256:dfded702919a2fcbaacd66e86a33ac3da643358b73f0e2524c175ac85bef8a6e%2findex.json" 1 {type application/vnd.docker.distribution.manifest.v1+json} {length 67494}} Content-Length: 531 Connection: close Content-Type: text/html; charset=iso-8859-1 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>406 Not Acceptable</title> </head><body> <h1>Not Acceptable</h1> <p>An appropriate representation of the requested resource could not be found on this server.</p> Available variants: <ul> <li><a href="sha256:dfded702919a2fcbaacd66e86a33ac3da643358b73f0e2524c175ac85bef8a6e%2findex.json">sha256:dfded702919a2fcbaacd66e86a33ac3da643358b73f0e2524c175ac85bef8a6e/index.json</a> , type application/vnd.docker.distribution.manifest.v1+json</li> </ul> </body></html> Version-Release number of selected component (if applicable): OSP 16.2.5 current How reproducible: Customer env and this lab; not sure how to reproduce Additional info: Will be provided.