Bug 1383816 - errors in registry logs
Summary: errors in registry logs
Keywords:
Status: CLOSED DUPLICATE of bug 1567658
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image Registry
Version: 3.2.0
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: ---
Assignee: Michal Minar
QA Contact: Wei Sun
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-10-11 20:35 UTC by Luke Meyer
Modified: 2023-09-14 03:32 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-04-26 14:53:43 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Luke Meyer 2016-10-11 20:35:15 UTC
Description of problem:
The docker image registry at a customer has several recurring errors reported, and it's not clear to me if they should really be considered "errors". I originally reported as https://github.com/openshift/docker-registry-extensions/issues/9 and https://github.com/openshift/docker-registry-extensions/issues/8 but since this is a registry on 3.2 I thought a bug might serve better.

In a customer environment with OSE 3.2 they're getting these errors in the registry logs:

       time="2016-10-10T22:01:04.451940836-04:00" level=error msg="response completed with error" err.code="BLOB_UNKNOWN" err.detail=sha256:398b653eab57fc5b408fdb65ab811ecd555d551da18e98400d14cb0d6249dfe9 err.message="blob unknown to registry" go.version=go1.4.2 http.request.host="172.30.248.117:5000" http.request.id=7956882c-26d4-4a2f-a8e7-aa0396622853 http.request.method=GET http.request.remoteaddr="10.1.2.1:58209" http.request.uri="/v2/orange/php/blobs/sha256:398b653eab57fc5b408fdb65ab811ecd555d551da18e98400d14cb0d6249dfe9" http.request.useragent="docker/1.10.3 go/go1.4.2 git-commit/7ffc8ee-unsupported kernel/3.10.0-327.13.1.el7.x86_64 os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=33.769186ms http.response.status=404 http.response.written=157 instance.id=a20494ee-c862-42a0-8893-577e8ef39ed8 vars.digest="sha256:398b653eab57fc5b408fdb65ab811ecd555d551da18e98400d14cb0d6249dfe9" vars.name="orange/php" 

I'm not sure what that means but it's in the logs a lot. Something asking about an image (layer) that's no longer in the registry?

       time="2016-10-10T22:01:42.623690508-04:00" level=error msg="Error retrieving ImageStreamImage admsdev/adms@sha256:43cf02f33d027880a33d27598cbb7b726ff445121a56872f2cc6d18ab8169a37: imagestreams \"adms\" not found" go.version=go1.4.2 http.request.host="172.30.248.117:5000" http.request.id=85ee9f32-bf9f-4653-be7d-49defa531542 http.request.method=GET http.request.remoteaddr="10.1.22.1:53215" http.request.uri="/v2/admsdev/adms/manifests/sha256:43cf02f33d027880a33d27598cbb7b726ff445121a56872f2cc6d18ab8169a37" http.request.useragent="docker/1.10.3 go/go1.4.2 git-commit/7ffc8ee-unsupported kernel/3.10.0-327.13.1.el7.x86_64 os/linux arch/amd64" instance.id=a20494ee-c862-42a0-8893-577e8ef39ed8 vars.name="admsdev/adms" vars.reference="sha256:43cf02f33d027880a33d27598cbb7b726ff445121a56872f2cc6d18ab8169a37" 
       time="2016-10-10T22:01:42.623884544-04:00" level=error msg="response completed with error" err.code="MANIFEST_UNKNOWN" err.detail="imagestreams \"adms\" not found" err.message="manifest unknown" go.version=go1.4.2 http.request.host="172.30.248.117:5000" http.request.id=85ee9f32-bf9f-4653-be7d-49defa531542 http.request.method=GET http.request.remoteaddr="10.1.22.1:53215" http.request.uri="/v2/admsdev/adms/manifests/sha256:43cf02f33d027880a33d27598cbb7b726ff445121a56872f2cc6d18ab8169a37" http.request.useragent="docker/1.10.3 go/go1.4.2 git-commit/7ffc8ee-unsupported kernel/3.10.0-327.13.1.el7.x86_64 os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=17.552851ms http.response.status=404 http.response.written=285 instance.id=a20494ee-c862-42a0-8893-577e8ef39ed8 vars.name="admsdev/adms" vars.reference="sha256:43cf02f33d027880a33d27598cbb7b726ff445121a56872f2cc6d18ab8169a37" 

Is this a simple "missing imagestream" message or does it point to something going wrong?

       time="2016-10-11T03:18:23.804470343-04:00" level=error msg="response completed with error" err.code="MANIFEST_UNKNOWN" err.detail="filesystem: Path not found: /docker/registry/v2/repositories/ctm-test/dispatcher-author/_manifests/revisions/sha256/599cb877ac19c22159fc0e3634fc456291f8a6862a6e4c04635d0cd9bcec034f/signatures/sha256" err.message="manifest unknown" go.version=go1.4.2 http.request.host="172.30.248.117:5000" http.request.id=c0a7b9ff-6152-49d4-835c-ccee415f5020 http.request.method=GET http.request.remoteaddr="10.1.6.1:40197" http.request.uri="/v2/ctm-test/dispatcher-author/manifests/sha256:599cb877ac19c22159fc0e3634fc456291f8a6862a6e4c04635d0cd9bcec034f" http.request.useragent="docker/1.10.3 go/go1.4.2 git-commit/7ffc8ee-unsupported kernel/3.10.0-327.13.1.el7.x86_64 os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=48.976977ms http.response.status=404 http.response.written=287 instance.id=c987c3e6-7bb7-43fa-aa16-d1be10d64c1e vars.name="ctm-test/dispatcher-author" vars.reference="sha256:599cb877ac19c22159fc0e3634fc456291f8a6862a6e4c04635d0cd9bcec034f" 

One that doesn't seem like it should be level=error is:

   time="2016-10-10T22:05:53.09732915-04:00" level=error msg="error authorizing context: authorization header with basic token required" go.version=go1.4.2 http.request.host="172.30.248.117:5000" http.request.id=dd935353-7781-4f6b-b042-066db87335ec http.request.method=GET http.request.remoteaddr="10.1.10.1:39413" http.request.uri="/v2/" http.request.useragent="docker/1.10.3 go/go1.4.2 git-commit/7ffc8ee-unsupported kernel/3.10.0-327.13.1.el7.x86_64 os/linux arch/amd64" instance.id=a20494ee-c862-42a0-8893-577e8ef39ed8 
   time="2016-10-10T22:07:26.553102691-04:00" level=error msg="error authorizing context: authorization header with basic token required" go.version=go1.4.2 http.request.host="172.30.248.117:5000" http.request.id=89f60a4d-8adb-4970-bc1c-13588bd17d3b http.request.method=GET http.request.remoteaddr="10.1.13.1:46212" http.request.uri="/v2/" http.request.useragent="docker/1.10.3 go/go1.4.2 git-commit/7ffc8ee-unsupported kernel/3.10.0-327.13.1.el7.x86_64 os/linux arch/amd64" instance.id=a20494ee-c862-42a0-8893-577e8ef39ed8 
   time="2016-10-10T22:07:56.468117847-04:00" level=error msg="error authorizing context: authorization header with basic token required" go.version=go1.4.2 http.request.host="172.30.248.117:5000" http.request.id=24165fad-ae4a-4632-8929-806ba6cf1685 http.request.method=GET http.request.remoteaddr="10.1.12.1:56036" http.request.uri="/v2/" http.request.useragent="docker/1.10.3 go/go1.4.2 git-commit/7ffc8ee-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64" instance.id=a20494ee-c862-42a0-8893-577e8ef39ed8 
   time="2016-10-10T22:11:50.090330658-04:00" level=error msg="error authorizing context: authorization header with basic token required" go.version=go1.4.2 http.request.host="172.30.248.117:5000" http.request.id=72c6db2b-c879-48e7-be75-b0df974ec4ff http.request.method=GET http.request.remoteaddr="10.1.10.1:39634" http.request.uri="/v2/" http.request.useragent="docker/1.10.3 go/go1.4.2 git-commit/7ffc8ee-unsupported kernel/3.10.0-327.13.1.el7.x86_64 os/linux arch/amd64" instance.id=a20494ee-c862-42a0-8893-577e8ef39ed8 
   time="2016-10-10T22:11:57.595052459-04:00" level=error msg="error authorizing context: authorization header with basic token required" go.version=go1.4.2 http.request.host="172.30.248.117:5000" http.request.id=f0d04ba8-3bba-40f4-8577-b1b27b72cd63 http.request.method=GET http.request.remoteaddr="10.1.22.1:54021" http.request.uri="/v2/" http.request.useragent="docker/1.10.3 go/go1.4.2 git-commit/7ffc8ee-unsupported kernel/3.10.0-327.13.1.el7.x86_64 os/linux arch/amd64" instance.id=a20494ee-c862-42a0-8893-577e8ef39ed8 

Diagnostics look through the registry logs for common errors and this doesn't seem like it really merits reporting to the user as an error.

If these aren't really errors in the operation of the registry, it would be nice if they could either be downgraded from level=error or if there were another level for tagging "real" errors.

Comment 1 Michal Fojtik 2016-10-12 08:58:41 UTC
These messages are not really an errors, just OpenShift requesting objects that does not really exists anymore in registry (because they were pruned/etc). They can be safely ignored and we will make a patch upstream that will change the log level.

Marking this as low priority (as it is not an error, just logging verbosity).

Comment 2 Luke Meyer 2016-10-12 20:00:08 UTC
One more just for completeness sake.

time="2016-09-29T15:16:19.557367666-04:00" level=error msg="Error getting ImageStreamTag \"latest\": imagestreamtags \"ctxmgr-app-dev:latest\" not found" go.version=go1.4.2 http.request.host="172.30.101.240:5000" http.request.id=d28759d7-ee48-4661-8158-7805819d3227 http.request.method=GET http.request.remoteaddr="10.1.11.1:54187" http.request.uri="/v2/linkctxmgrdev/ctxmgr-app-dev/manifests/latest" http.request.useragent="docker/1.10.3 go/go1.4.2 git-commit/7ffc8ee-unsupported kernel/3.10.0-327.13.1.el7.x86_64 os/linux arch/amd64" instance.id=0294efca-bce9-45dd-9864-13824c318bec vars.name="linkctxmgrdev/ctxmgr-app-dev" vars.reference=latest

Comment 3 Ben Parees 2017-11-06 23:36:53 UTC
Michal, did that upstream patch ever happen?

Comment 4 Ben Parees 2018-01-10 04:31:57 UTC
Michal, bump.

Comment 5 Michal Minar 2018-04-26 14:53:43 UTC
> Michal, did that upstream patch ever happen?

I'm afraid not.

Closing this one even though it's older since the newer bug has more up-to-date info and idieas on resolution.

*** This bug has been marked as a duplicate of bug 1567658 ***

Comment 6 Red Hat Bugzilla 2023-09-14 03:32:18 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days


Note You need to log in before you can comment on or make changes to this bug.