Hide Forgot
Description of problem: Tags in ImageStream configured with importPolicy.scheduled: true are not updated when they reference a different image in remote repository Version-Release number of selected component (if applicable): How reproducible: Always Steps to Reproduce: 1. Create an ImageStream object as such: apiVersion: v1 kind: ImageStream metadata: annotations: openshift.io/image.insecureRepository: "true" name: lucarval-rsyslog spec: dockerImageRepository: 172.17.0.1:8080/redhat-lucarval-rsyslog tags: - annotations: null from: kind: DockerImage name: 172.17.0.1:8080/redhat-lucarval-rsyslog:latest importPolicy: insecure: true scheduled: true name: latest 2. Verify image info is successfully retrieved: oc get imagestreamtag lucarval-rsyslog:latest 3. Push a new image to registry to change latest tag Actual results: After configured scheduledImageImportMinimumIntervalSeconds has elapsed, ImageStreamTag still references previous image: oc get imagestreamtag lucarval-rsyslog:latest Log displays the following error: time="2016-12-09T14:31:22Z" level=error msg="error adding descriptor sha256:821a26b9542a1eb5c575a9f5095cb5c652a5f05b14b17d77e661ae1df7003a2b to cache: cache: empty mediatype on descriptor: { 213288960 sha256:821a26b9542a1eb5c575a9f5095cb5c652a5f05b14b17d77e661ae1df7003a2b []}" go.version=go1.6.3 Expected results: After configured scheduledImageImportMinimumIntervalSeconds has elapsed, ImageStreamTag should reference a different image: oc get imagestreamtag lucarval-rsyslog:latest Additional info: This seems to be related to missing Content-Type header in response from registry. The registry in use is Crane. Although, the registry should set the Content-Type header properly, this behavior is unusual because OpenShift is able to fetch the images initially, it's the scheduled check that fails. Also, this registry works fine with docker pull.
It seems that having any "Content-Type" at all is sufficient to bypass this issue. For example, if "Content-Type" is set to "text/plain", even when it should really be something along these lines "application/vnd.docker.distribution.manifest.v1+prettyjws", OpenShift seems to fetch updated tags properly. This is the case for our customer facing registry: # curl -ILk https://registry.access.redhat.com/v2/rhel7/manifests/latestt HTTP/1.1 302 FOUND Date: Mon, 12 Dec 2016 14:59:05 GMT Server: Apache Content-Length: 461 Location: https://access.redhat.com/webassets/docker/content/dist/rhel/server/7/7Server/x86_64/containers/registry/rhel7/manifests/latest Connection: close Content-Type: text/html; charset=utf-8 HTTP/1.1 200 OK Server: Apache ETag: "115f3456bc5e47cf35b6f6e37339cb5e:1481038620" Last-Modified: Tue, 06 Dec 2016 15:31:48 GMT Accept-Ranges: bytes Content-Length: 5946 Content-Type: text/plain X-Docker-Size: -1 Date: Mon, 12 Dec 2016 14:59:06 GMT Connection: keep-alive X-Docker-Size: -1
The error you're seeing is causing OpenShift not to cache [1] information from registry but reach out to registry each time, it's not a problem per se and it is not related to any failures you're seeing, since no error is being returned at that point (see attached piece of code). I've verified this on my local environment and the error is since the first hit to registry. It would be nice, though to set appropriate Content-Type being returned by the crane registry. [1] https://github.com/openshift/origin/blob/eb304fddd41da78c0926eb965f1a8a02589b1b85/vendor/github.com/docker/distribution/registry/storage/cache/cachedblobdescriptorstore.go#L98
The issue reported here is that imagePolicy.scheduled does not work at all in the case where Content-Type is not set in the response. I agree the correct way to solve this issue is by actually setting the Content-Type. I'd like to point out again that this is an inconsistent behavior in OpenShift though since it is able to update the ImageStream correctly upon its creation and by using oc import-image. In addition, docker daemon has no issues processing registries that omit the Content-Type. Perhaps, requiring Content-Type in response can be documented somewhere?
I've checked with latest master [1] that it's working as expected. The error you're seeing is not causing any problems, just informing (in nasty way, that I agree) that it can't cache information due to aforementioned problems. So it ends up reaching registry each time it needs to fetch that information. But scheduled import is being invoked as designed. https://github.com/openshift/origin/commit/eb304fddd41da78c0926eb965f1a8a02589b1b85
The scheduled import does happen as you pointed out. However, the new image is **not** imported. There's something wrong in this behavior.
With most recent version of pulp, content-type headers are set in response, which makes this no longer an issue.