Bug 1418358 - [3.3] Image data stored in ETCD accounting for large DB.
Summary: [3.3] Image data stored in ETCD accounting for large DB.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image Registry
Version: 3.3.1
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 3.3.1
Assignee: Michal Minar
QA Contact: ge liu
URL:
Whiteboard:
: 1378180 1435536 (view as bug list)
Depends On: 1378180 1418359
Blocks: 1267746
TreeView+ depends on / blocked
 
Reported: 2017-02-01 16:05 UTC by Michal Minar
Modified: 2021-08-30 13:15 UTC (History)
19 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: OpenShift cluster used to store manifests of all images in etcd database. Consequence: The manifests occupied a lot of space in the database making it big and slow. Fix: The integrated registry now stores manifests in its associated storage rather than in etcd. Also manifests of remote images are not stored at all. They are fetch from external registries when needed. A migration script has been provided to move manifests from all existing images in the cluster into the integrated registry. Result: Newly pushed images will not cause etcd database to grow so fast. By using the migration script, the admin is able to reduce etcd size considerably.
Clone Of: 1378180
Environment:
Last Closed: 2017-06-15 18:36:44 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Origin (Github) 11925 0 None None None 2017-02-01 16:05:02 UTC
Red Hat Knowledge Base (Solution) 2688911 0 None None None 2017-02-01 16:05:02 UTC
Red Hat Product Errata RHBA-2017:1425 0 normal SHIPPED_LIVE OpenShift Container Platform 3.5, 3.4, 3.3, and 3.2 bug fix update 2017-06-15 22:35:53 UTC

Comment 2 ge liu 2017-02-09 10:09:47 UTC
Verified with fork_ami:

# openshift version
openshift v1.3.0-alpha.3+de923b4
kubernetes v1.3.0+52492b4
etcd 2.3.0+git


# oc import-image --confirm --from=centos/ruby-22-centos7 ruby-22-centos7:latest
The import completed successfully.

# oc get images -o jsonpath=$'{.dockerImageManifest}\n' sha256:460730627b1c32c10368aa754e7075e447394b76c5a213a819a11297000edd2d

[root@ip-172-18-xx-xxx ~]

# oc new-build --image-stream=ruby-22-centos7 https://github.com/openshift/ruby-ex.git


# curl -u geliu:M3tYQ3jH0SM0fznV9Lucr5hCReWKFW2PN0RI9TEVWOw http://172.xx.xx.xxx:5000/v2/lgrpoj/ruby-ex/manifests/latest
{
   "schemaVersion": 1,
   "name": "lgrpoj/ruby-ex",
   "tag": "latest",
   "architecture": "amd64",
   "fsLayers": [
      {
         "blobSum": "sha256:3324098f4ed7bd31e56210882838a0546cfd08211c28e3853e278d9d63f77a49"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:2be9ff29c58d38b6a4683af6dc6d90a16086b360bd058677923844462a617449"
      },
      {
         "blobSum": "sha256:f49e17d19b2d4a2e9d75cab6c0398bc617245edfdeeeb2e6949376f2b84a1593"
      },
      {
         "blobSum": "sha256:4451112364bfed82a4e7f02ae16b1e8fe5c5a69d18c906cee29a005a8de4a4d8"
      },
      {
         "blobSum": "sha256:3c0cc59b158068ea21588410fee4323031ff547905b4e4b1758755c8e5fc84f5"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:8fbef6a4730a0f02f60311a4b54ff44deeed9e43d9815cf89c779e49e617faf1"
      },
      {
         "blobSum": "sha256:d6d897d19b25daf0149bb3b1f70f9eb3c0b35df82db95e8999a8802e7136630b"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:3f92963ff97b7ecef976010b5e1d257ca91ecf8a6bc70542c6dc4540b587ed7b"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:45a2e645736c4c66ef34acce2407ded21f7a9b231199d3b92d6c9776df264729"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      }
   ],
   "history": [
      {
         "v1Compatibility": "{\"architecture\":\"amd64\",\"config\":{\"Hostname\":\"\",\"Domainname\":\"\",\"User\":\"1001\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"ExposedPorts\":{\"8080/tcp\":{}},\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"OPENSHIFT_BUILD_NAME=ruby-ex-1\",\"OPENSHIFT_BUILD_NAMESPACE=lgrpoj\",\"OPENSHIFT_BUILD_SOURCE=https://github.com/openshift/ruby-ex.git\",\"PATH=/opt/app-root/src/bin:/opt/app-root/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\",\"STI_SCRIPTS_URL=image:///usr/libexec/s2i\",\"STI_SCRIPTS_PATH=/usr/libexec/s2i\",\"HOME=/opt/app-root/src\",\"BASH_ENV=/opt/app-root/etc/scl_enable\",\"ENV=/opt/app-root/etc/scl_enable\",\"PROMPT_COMMAND=. /opt/app-root/etc/scl_enable\",\"RUBY_VERSION=2.2\"],\"Cmd\":[\"/usr/libexec/s2i/run\"],\"Image\":\"\",\"Volumes\":null,\"WorkingDir\":\"/opt/app-root/src\",\"Entrypoint\":[\"container-entrypoint\"],\"OnBuild\":null,\"Labels\":{\"build-date\":\"20161214\",\"io.k8s.description\":\"Platform for building and running Ruby 2.2 applications\",\"io.k8s.display-name\":\"lgrpoj/ruby-ex-1:5ed595da\",\"io.openshift.build.commit.author\":\"Ionut Palade \\u003cPI-Victor.github.com\\u003e\",\"io.openshift.build.commit.date\":\"Mon Dec 12 14:37:32 2016 +0100\",\"io.openshift.build.commit.id\":\"855ab2de53ff897a19e1055f7554c64d19e02c50\",\"io.openshift.build.commit.message\":\"Merge pull request #6 from aj07/typo\",\"io.openshift.build.commit.ref\":\"master\",\"io.openshift.build.image\":\"centos/ruby-22-centos7@sha256:460730627b1c32c10368aa754e7075e447394b76c5a213a819a11297000edd2d\",\"io.openshift.build.source-location\":\"https://github.com/openshift/ruby-ex.git\",\"io.openshift.builder-base-version\":\"bfd4736\",\"io.openshift.builder-version\":\"06c0ec324aa6edf151f4ea1e7304199c72011bec\",\"io.openshift.expose-services\":\"8080:http\",\"io.openshift.s2i.scripts-url\":\"image:///usr/libexec/s2i\",\"io.openshift.tags\":\"builder,ruby,ruby22\",\"io.s2i.scripts-url\":\"image:///usr/libexec/s2i\",\"license\":\"GPLv2\",\"name\":\"CentOS Base Image\",\"vendor\":\"CentOS\"}},\"container\":\"3076ad6ef96a404faccddc784cc54d1a78d8eaf453f51d61977a1ce15d528c29\",\"container_config\":{\"Hostname\":\"ruby-ex-1-build\",\"Domainname\":\"\",\"User\":\"1001\",\"AttachStdin\":false,\"AttachStdout\":true,\"AttachStderr\":false,\"ExposedPorts\":{\"8080/tcp\":{}},\"Tty\":false,\"OpenStdin\":true,\"StdinOnce\":true,\"Env\":[\"OPENSHIFT_BUILD_NAME=ruby-ex-1\",\"OPENSHIFT_BUILD_NAMESPACE=lgrpoj\",\"OPENSHIFT_BUILD_SOURCE=https://github.com/openshift/ruby-ex.git\",\"PATH=/opt/app-root/src/bin:/opt/app-root/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\",\"STI_SCRIPTS_URL=image:///usr/libexec/s2i\",\"STI_SCRIPTS_PATH=/usr/libexec/s2i\",\"HOME=/opt/app-root/src\",\"BASH_ENV=/opt/app-root/etc/scl_enable\",\"ENV=/opt/app-root/etc/scl_enable\",\"PROMPT_COMMAND=. /opt/app-root/etc/scl_enable\",\"RUBY_VERSION=2.2\"],\"Cmd\":[\"/bin/sh\",\"-c\",\"tar -C /tmp -xf - \\u0026\\u0026 /usr/libexec/s2i/assemble\"],\"Image\":\"centos/ruby-22-centos7@sha256:460730627b1c32c10368aa754e7075e447394b76c5a213a819a11297000edd2d\",\"Volumes\":null,\"WorkingDir\":\"/opt/app-root/src\",\"Entrypoint\":[\"/bin/env\"],\"OnBuild\":null,\"Labels\":{\"build-date\":\"20161214\",\"io.k8s.description\":\"Platform for building and running Ruby 2.2 applications\",\"io.k8s.display-name\":\"lgrpoj/ruby-ex-1:5ed595da\",\"io.openshift.build.commit.author\":\"Ionut Palade \\u003cPI-Victor.github.com\\u003e\",\"io.openshift.build.commit.date\":\"Mon Dec 12 14:37:32 2016 +0100\",\"io.openshift.build.commit.id\":\"855ab2de53ff897a19e1055f7554c64d19e02c50\",\"io.openshift.build.commit.message\":\"Merge pull request #6 from aj07/typo\",\"io.openshift.build.commit.ref\":\"master\",\"io.openshift.build.image\":\"centos/ruby-22-centos7@sha256:460730627b1c32c10368aa754e7075e447394b76c5a213a819a11297000edd2d\",\"io.openshift.build.source-location\":\"https://github.com/openshift/ruby-ex.git\",\"io.openshift.builder-base-version\":\"bfd4736\",\"io.openshift.builder-version\":\"06c0ec324aa6edf151f4ea1e7304199c72011bec\",\"io.openshift.expose-services\":\"8080:http\",\"io.openshift.s2i.scripts-url\":\"image:///usr/libexec/s2i\",\"io.openshift.tags\":\"builder,ruby,ruby22\",\"io.s2i.scripts-url\":\"image:///usr/libexec/s2i\",\"license\":\"GPLv2\",\"name\":\"CentOS Base Image\",\"vendor\":\"CentOS\"}},\"created\":\"2017-02-09T10:01:39.607932417Z\",\"docker_version\":\"1.10.3\",\"id\":\"2544e3a42489d71f7fb05328d8b5bcb7b0ed3f00d2418012d514a794b16e6f7b\",\"os\":\"linux\",\"parent\":\"62a557a72493406068264efccf4a03a6b15446d1a114af55f0756f997f60e7ec\"}"
      },
      {
         "v1Compatibility": "{\"id\":\"62a557a72493406068264efccf4a03a6b15446d1a114af55f0756f997f60e7ec\",\"parent\":\"5f61032c60977ebb9f35ae87289fdaed21de0c6f2cb2eaf9b803a116c051c79a\",\"created\":\"2017-02-07T16:23:34.396064385Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) LABEL io.openshift.builder-version=06c0ec324aa6edf151f4ea1e7304199c72011bec\"]},\"author\":\"SoftwareCollections.org \\u003csclorg\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"5f61032c60977ebb9f35ae87289fdaed21de0c6f2cb2eaf9b803a116c051c79a\",\"parent\":\"3f30f7308b8edb50f6fdd058bd7d9c504445538c94c02ede67f65a197941044c\",\"created\":\"2017-02-07T16:23:26.981260791Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) CMD [\\\"/bin/sh\\\" \\\"-c\\\" \\\"$STI_SCRIPTS_PATH/usage\\\"]\"]},\"author\":\"SoftwareCollections.org \\u003csclorg\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"3f30f7308b8edb50f6fdd058bd7d9c504445538c94c02ede67f65a197941044c\",\"parent\":\"f6361783d5bd16bafd68299087d6fb0e7fb9a66299f9267c998b9fc379e3af80\",\"created\":\"2017-02-07T16:23:19.402056901Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) USER [1001]\"]},\"author\":\"SoftwareCollections.org \\u003csclorg\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"f6361783d5bd16bafd68299087d6fb0e7fb9a66299f9267c998b9fc379e3af80\",\"parent\":\"721be1a2a1d46246b5f265f833744eb47532b3f9c198e7b623a7dcc6d12c5362\",\"created\":\"2017-02-07T16:23:09.117473874Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c chown -R 1001:0 /opt/app-root \\u0026\\u0026 chmod -R ug+rwx /opt/app-root\"]},\"author\":\"SoftwareCollections.org \\u003csclorg\\u003e\"}"
      },
      {
         "v1Compatibility": "{\"id\":\"721be1a2a1d46246b5f265f833744eb47532b3f9c198e7b623a7dcc6d12c5362\",\"parent\":\"a50f754e2c44ae39e62ab368a9230801c44fcf2b512b6db157432f467b97c339\",\"created\":\"2017-02-07T16:23:00.355368979Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) COPY dir:4abb8aa4037d4ccf9187e54dddd8fd759645a008de59c2ae03f09f6c510a90a9 in /opt/app-root\"]},\"author\":\"SoftwareCollections.org \\u003csclorg\\u003e\"}"
      },
      {
         "v1Compatibility": "{\"id\":\"a50f754e2c44ae39e62ab368a9230801c44fcf2b512b6db157432f467b97c339\",\"parent\":\"d53f1df78daf1e1fe406b75d0ea1cb128e1bbef1d9587a9aa8c85cf9f61c1a10\",\"created\":\"2017-02-07T16:22:47.236542373Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) COPY dir:b1da86c0f4ca04fed6e7bd8c3fa3c9896deba393695ed3c753a8a1a088f60c12 in /usr/libexec/s2i\"]},\"author\":\"SoftwareCollections.org \\u003csclorg\\u003e\"}"
      },
      {
         "v1Compatibility": "{\"id\":\"d53f1df78daf1e1fe406b75d0ea1cb128e1bbef1d9587a9aa8c85cf9f61c1a10\",\"parent\":\"118464f0bb32ab79be2bbb1d6a41319e939316f0bdcd6ab0705ede7cc9a951a6\",\"created\":\"2017-02-07T16:22:30.530473423Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c INSTALL_PKGS=\\\"rh-ruby22 rh-ruby22-ruby-devel rh-ruby22-rubygem-rake v8314 rh-ruby22-rubygem-bundler nodejs010\\\" \\u0026\\u0026     yum install -y centos-release-scl \\u0026\\u0026     yum install -y --setopt=tsflags=nodocs  $INSTALL_PKGS \\u0026\\u0026 rpm -V $INSTALL_PKGS \\u0026\\u0026     yum clean all -y\"]},\"author\":\"SoftwareCollections.org \\u003csclorg\\u003e\"}"
      },
      {
         "v1Compatibility": "{\"id\":\"118464f0bb32ab79be2bbb1d6a41319e939316f0bdcd6ab0705ede7cc9a951a6\",\"parent\":\"57f6f68f2c2c016843b9c1b794962413ae371c98d385e8fe3f69b4044184b6fa\",\"created\":\"2017-02-07T16:19:30.06429988Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) LABEL io.k8s.description=Platform for building and running Ruby 2.2 applications io.k8s.display-name=Ruby 2.2 io.openshift.expose-services=8080:http io.openshift.tags=builder,ruby,ruby22\"]},\"author\":\"SoftwareCollections.org \\u003csclorg\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"57f6f68f2c2c016843b9c1b794962413ae371c98d385e8fe3f69b4044184b6fa\",\"parent\":\"206c0879d9604c8dd2bf46750f00f4cfbd06986dcba5c556715e45562adce70f\",\"created\":\"2017-02-07T16:19:24.434230875Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) ENV RUBY_VERSION=2.2\"]},\"author\":\"SoftwareCollections.org \\u003csclorg\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"206c0879d9604c8dd2bf46750f00f4cfbd06986dcba5c556715e45562adce70f\",\"parent\":\"b00834916802b3d98f480b9cecaa0ffba9fb5591608574175fc07733485a2cfd\",\"created\":\"2017-02-07T16:19:18.827146476Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) EXPOSE 8080/tcp\"]},\"author\":\"SoftwareCollections.org \\u003csclorg\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"b00834916802b3d98f480b9cecaa0ffba9fb5591608574175fc07733485a2cfd\",\"parent\":\"6366dab18770823fb79819a2fbc42834ccdae91046366c539c5772ce334620ed\",\"created\":\"2017-02-07T16:19:11.947245564Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) MAINTAINER SoftwareCollections.org \\u003csclorg\\u003e\"]},\"author\":\"SoftwareCollections.org \\u003csclorg\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"6366dab18770823fb79819a2fbc42834ccdae91046366c539c5772ce334620ed\",\"parent\":\"88dec2b607ddf20dd46cc8839e59abafe25eba8b1df10e7e3e09a03143775ba6\",\"created\":\"2017-02-07T14:40:06.888994193Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) LABEL io.openshift.builder-base-version=bfd4736\"]},\"author\":\"Jakub Hadvig \\u003cjhadvig\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"88dec2b607ddf20dd46cc8839e59abafe25eba8b1df10e7e3e09a03143775ba6\",\"parent\":\"86e1c27719bdd532e3c37323e52ad78d47cb3aa6eea83aa382fa4ed99b6a4839\",\"created\":\"2017-02-07T14:40:02.194720807Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) CMD [\\\"base-usage\\\"]\"]},\"author\":\"Jakub Hadvig \\u003cjhadvig\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"86e1c27719bdd532e3c37323e52ad78d47cb3aa6eea83aa382fa4ed99b6a4839\",\"parent\":\"86bf6d084a80df2eb177d209a12a81022ca28ce2eb1963356e88b5fd8ea40698\",\"created\":\"2017-02-07T14:39:57.866183301Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) ENTRYPOINT \\u0026{[\\\"container-entrypoint\\\"]}\"]},\"author\":\"Jakub Hadvig \\u003cjhadvig\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"86bf6d084a80df2eb177d209a12a81022ca28ce2eb1963356e88b5fd8ea40698\",\"parent\":\"8d15ff5d1ef88afe37983b8e302294174e2de965b1abe6e326d928e5df6156b3\",\"created\":\"2017-02-07T14:39:51.919682393Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) WORKDIR /opt/app-root/src\"]},\"author\":\"Jakub Hadvig \\u003cjhadvig\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"8d15ff5d1ef88afe37983b8e302294174e2de965b1abe6e326d928e5df6156b3\",\"parent\":\"a3086aa2a7b909ad3f96307e7f400a5f1e1acec2c8ab425724659a518ac0a511\",\"created\":\"2017-02-07T14:39:48.171094843Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) COPY dir:7c2ced3187c8deac6a4ede38feda6eaacbef85bf4975d7979aea4b79db005ca8 in /usr/bin/\"]},\"author\":\"Jakub Hadvig \\u003cjhadvig\\u003e\"}"
      },
      {
         "v1Compatibility": "{\"id\":\"a3086aa2a7b909ad3f96307e7f400a5f1e1acec2c8ab425724659a518ac0a511\",\"parent\":\"4bf4f26f8082668cd81b45378622ef918ca73edf60e84ae542c09732656b55f3\",\"created\":\"2016-12-20T10:00:15.084189736Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c rpmkeys --import file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7 \\u0026\\u0026   INSTALL_PKGS=\\\"autoconf   automake   bsdtar   bzip2   findutils   gcc-c++   gd-devel   gdb   gettext   git   libcurl-devel   libxml2-devel   libxslt-devel   lsof   make   mariadb-devel   mariadb-libs   openssl-devel   patch   postgresql-devel   procps-ng   scl-utils   sqlite-devel   tar   unzip   wget   which   yum-utils   zlib-devel\\\" \\u0026\\u0026   mkdir -p ${HOME}/.pki/nssdb \\u0026\\u0026   chown -R 1001:0 ${HOME}/.pki \\u0026\\u0026   yum install -y --setopt=tsflags=nodocs $INSTALL_PKGS \\u0026\\u0026   rpm -V $INSTALL_PKGS \\u0026\\u0026   yum clean all -y \\u0026\\u0026   useradd -u 1001 -r -g 0 -d ${HOME} -s /sbin/nologin       -c \\\"Default Application User\\\" default \\u0026\\u0026   chown -R 1001:0 /opt/app-root\"]},\"author\":\"Jakub Hadvig \\u003cjhadvig\\u003e\"}"
      },
      {
         "v1Compatibility": "{\"id\":\"4bf4f26f8082668cd81b45378622ef918ca73edf60e84ae542c09732656b55f3\",\"parent\":\"6f2595bffdc30a81ae41b5e2244e9ec46a25951842349002da5186243253d17a\",\"created\":\"2016-12-20T09:57:33.884461265Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) ENV BASH_ENV=/opt/app-root/etc/scl_enable ENV=/opt/app-root/etc/scl_enable PROMPT_COMMAND=. /opt/app-root/etc/scl_enable\"]},\"author\":\"Jakub Hadvig \\u003cjhadvig\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"6f2595bffdc30a81ae41b5e2244e9ec46a25951842349002da5186243253d17a\",\"parent\":\"b7a9f751f2d64842b35f56eebd2f87929aa4e34d440cd551b394344972b418ba\",\"created\":\"2016-12-20T09:57:13.815581712Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) COPY file:ebe723f29ba98a751f6afca41fea401f2c20174f413ab2a7eb4eaeaa9a322cc7 in /opt/app-root/etc/scl_enable\"]},\"author\":\"Jakub Hadvig \\u003cjhadvig\\u003e\"}"
      },
      {
         "v1Compatibility": "{\"id\":\"b7a9f751f2d64842b35f56eebd2f87929aa4e34d440cd551b394344972b418ba\",\"parent\":\"e78b2fb0ee8d9af79e3dae18cc9702c087ca9de69ae071dbb74a27becf4bf07a\",\"created\":\"2016-12-20T09:57:01.090279875Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) ENV STI_SCRIPTS_URL=image:///usr/libexec/s2i STI_SCRIPTS_PATH=/usr/libexec/s2i HOME=/opt/app-root/src PATH=/opt/app-root/src/bin:/opt/app-root/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"]},\"author\":\"Jakub Hadvig \\u003cjhadvig\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"e78b2fb0ee8d9af79e3dae18cc9702c087ca9de69ae071dbb74a27becf4bf07a\",\"parent\":\"4ef929f1a369bd03c4c91e32b0e6d62c0fbdae92a7563aeb657b3102ade0f62a\",\"created\":\"2016-12-20T09:56:43.287284059Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) LABEL io.openshift.s2i.scripts-url=image:///usr/libexec/s2i io.s2i.scripts-url=image:///usr/libexec/s2i\"]},\"author\":\"Jakub Hadvig \\u003cjhadvig\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"4ef929f1a369bd03c4c91e32b0e6d62c0fbdae92a7563aeb657b3102ade0f62a\",\"parent\":\"08e1d80f2b80b86e13801c44c691fbc5812755972d44c1b2942f7f0cfa00a438\",\"created\":\"2016-12-20T09:56:30.945436702Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) MAINTAINER Jakub Hadvig \\u003cjhadvig\\u003e\"]},\"author\":\"Jakub Hadvig \\u003cjhadvig\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"08e1d80f2b80b86e13801c44c691fbc5812755972d44c1b2942f7f0cfa00a438\",\"parent\":\"a04895de19968b0665115154295009f75bcba85eb71dbb7fe6ae14cb8c95a475\",\"created\":\"2016-12-15T18:21:23.875991987Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop)  CMD [\\\"/bin/bash\\\"]\"]},\"author\":\"https://github.com/CentOS/sig-cloud-instance-images\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"a04895de19968b0665115154295009f75bcba85eb71dbb7fe6ae14cb8c95a475\",\"parent\":\"0a444b299d5a7868b640b12020a2684a72167b6710037b6bd4933f6713b2b47c\",\"created\":\"2016-12-15T18:21:23.00781987Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop)  LABEL name=CentOS Base Image vendor=CentOS license=GPLv2 build-date=20161214\"]},\"author\":\"https://github.com/CentOS/sig-cloud-instance-images\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"0a444b299d5a7868b640b12020a2684a72167b6710037b6bd4933f6713b2b47c\",\"parent\":\"3690474eb5b4b26fdfbd89c6e159e8cc376ca76ef48032a30fa6aafd56337880\",\"created\":\"2016-12-15T18:21:21.197722058Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) ADD file:940c77b6724c00d4208cc72169a63951eaa605672bcc5902ab2013cbae107434 in / \"]},\"author\":\"https://github.com/CentOS/sig-cloud-instance-images\"}"
      },
      {
         "v1Compatibility": "{\"id\":\"3690474eb5b4b26fdfbd89c6e159e8cc376ca76ef48032a30fa6aafd56337880\",\"created\":\"2016-08-30T18:18:45.109943798Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop)  MAINTAINER https://github.com/CentOS/sig-cloud-instance-images\"]},\"author\":\"https://github.com/CentOS/sig-cloud-instance-images\",\"throwaway\":true}"
      }
   ],
   "signatures": [
      {
         "header": {
            "jwk": {
               "crv": "P-256",
               "kid": "HWTR:AWP6:JIJF:VIQT:5RR2:DM7W:T3ED:47JU:D5C6:M7JY:5HQJ:C5CV",
               "kty": "EC",
               "x": "x45hKKUB7gB98oZJod5FA6W3mqF49kfiTol5F1Rn8WE",
               "y": "b9lpc4XHhhrPv1kfFXBvWj4WBVXh2N2DkIT3VIhhzvQ"
            },
            "alg": "ES256"
         },
         "signature": "NuAQnUGV4kjrepkj5bMCiJVctpFSenMw-OHDfIMNxzIs6nLFxwGjcVjBv4RUYvbMXxy_PtCS8_xmbF5OcaHofQ",
         "protected": "eyJmb3JtYXRMZW5ndGgiOjIwNzM5LCJmb3JtYXRUYWlsIjoiQ24wIiwidGltZSI6IjIwMTctMDItMDlUMTA6MDI6MjNaIn0"
      }
   ]
}

# curl -u geliu:M3tYQ3jH0SM0fznV9Lucr5hCReWKFW2PN0RI9TEVWOw http://172.xx.xx.xx:5000/v2/lgrpoj/ruby-ex/manifests/sha256:bb79b064434b859bdc5de90d2c3e68704b19ef018b4dc5d393a1509940375b0a
{
   "schemaVersion": 1,
   "name": "lgrpoj/ruby-ex",
   "tag": "latest",
   "architecture": "amd64",
   "fsLayers": [
      {
         "blobSum": "sha256:3324098f4ed7bd31e56210882838a0546cfd08211c28e3853e278d9d63f77a49"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:2be9ff29c58d38b6a4683af6dc6d90a16086b360bd058677923844462a617449"
      },
      {
         "blobSum": "sha256:f49e17d19b2d4a2e9d75cab6c0398bc617245edfdeeeb2e6949376f2b84a1593"
      },
      {
         "blobSum": "sha256:4451112364bfed82a4e7f02ae16b1e8fe5c5a69d18c906cee29a005a8de4a4d8"
      },
      {
         "blobSum": "sha256:3c0cc59b158068ea21588410fee4323031ff547905b4e4b1758755c8e5fc84f5"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:8fbef6a4730a0f02f60311a4b54ff44deeed9e43d9815cf89c779e49e617faf1"
      },
      {
         "blobSum": "sha256:d6d897d19b25daf0149bb3b1f70f9eb3c0b35df82db95e8999a8802e7136630b"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:3f92963ff97b7ecef976010b5e1d257ca91ecf8a6bc70542c6dc4540b587ed7b"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:45a2e645736c4c66ef34acce2407ded21f7a9b231199d3b92d6c9776df264729"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      }
   ],
   "history": [
      {
         "v1Compatibility": "{\"architecture\":\"amd64\",\"config\":{\"Hostname\":\"\",\"Domainname\":\"\",\"User\":\"1001\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"ExposedPorts\":{\"8080/tcp\":{}},\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"OPENSHIFT_BUILD_NAME=ruby-ex-1\",\"OPENSHIFT_BUILD_NAMESPACE=lgrpoj\",\"OPENSHIFT_BUILD_SOURCE=https://github.com/openshift/ruby-ex.git\",\"PATH=/opt/app-root/src/bin:/opt/app-root/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\",\"STI_SCRIPTS_URL=image:///usr/libexec/s2i\",\"STI_SCRIPTS_PATH=/usr/libexec/s2i\",\"HOME=/opt/app-root/src\",\"BASH_ENV=/opt/app-root/etc/scl_enable\",\"ENV=/opt/app-root/etc/scl_enable\",\"PROMPT_COMMAND=. /opt/app-root/etc/scl_enable\",\"RUBY_VERSION=2.2\"],\"Cmd\":[\"/usr/libexec/s2i/run\"],\"Image\":\"\",\"Volumes\":null,\"WorkingDir\":\"/opt/app-root/src\",\"Entrypoint\":[\"container-entrypoint\"],\"OnBuild\":null,\"Labels\":{\"build-date\":\"20161214\",\"io.k8s.description\":\"Platform for building and running Ruby 2.2 applications\",\"io.k8s.display-name\":\"lgrpoj/ruby-ex-1:5ed595da\",\"io.openshift.build.commit.author\":\"Ionut Palade \\u003cPI-Victor.github.com\\u003e\",\"io.openshift.build.commit.date\":\"Mon Dec 12 14:37:32 2016 +0100\",\"io.openshift.build.commit.id\":\"855ab2de53ff897a19e1055f7554c64d19e02c50\",\"io.openshift.build.commit.message\":\"Merge pull request #6 from aj07/typo\",\"io.openshift.build.commit.ref\":\"master\",\"io.openshift.build.image\":\"centos/ruby-22-centos7@sha256:460730627b1c32c10368aa754e7075e447394b76c5a213a819a11297000edd2d\",\"io.openshift.build.source-location\":\"https://github.com/openshift/ruby-ex.git\",\"io.openshift.builder-base-version\":\"bfd4736\",\"io.openshift.builder-version\":\"06c0ec324aa6edf151f4ea1e7304199c72011bec\",\"io.openshift.expose-services\":\"8080:http\",\"io.openshift.s2i.scripts-url\":\"image:///usr/libexec/s2i\",\"io.openshift.tags\":\"builder,ruby,ruby22\",\"io.s2i.scripts-url\":\"image:///usr/libexec/s2i\",\"license\":\"GPLv2\",\"name\":\"CentOS Base Image\",\"vendor\":\"CentOS\"}},\"container\":\"3076ad6ef96a404faccddc784cc54d1a78d8eaf453f51d61977a1ce15d528c29\",\"container_config\":{\"Hostname\":\"ruby-ex-1-build\",\"Domainname\":\"\",\"User\":\"1001\",\"AttachStdin\":false,\"AttachStdout\":true,\"AttachStderr\":false,\"ExposedPorts\":{\"8080/tcp\":{}},\"Tty\":false,\"OpenStdin\":true,\"StdinOnce\":true,\"Env\":[\"OPENSHIFT_BUILD_NAME=ruby-ex-1\",\"OPENSHIFT_BUILD_NAMESPACE=lgrpoj\",\"OPENSHIFT_BUILD_SOURCE=https://github.com/openshift/ruby-ex.git\",\"PATH=/opt/app-root/src/bin:/opt/app-root/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\",\"STI_SCRIPTS_URL=image:///usr/libexec/s2i\",\"STI_SCRIPTS_PATH=/usr/libexec/s2i\",\"HOME=/opt/app-root/src\",\"BASH_ENV=/opt/app-root/etc/scl_enable\",\"ENV=/opt/app-root/etc/scl_enable\",\"PROMPT_COMMAND=. /opt/app-root/etc/scl_enable\",\"RUBY_VERSION=2.2\"],\"Cmd\":[\"/bin/sh\",\"-c\",\"tar -C /tmp -xf - \\u0026\\u0026 /usr/libexec/s2i/assemble\"],\"Image\":\"centos/ruby-22-centos7@sha256:460730627b1c32c10368aa754e7075e447394b76c5a213a819a11297000edd2d\",\"Volumes\":null,\"WorkingDir\":\"/opt/app-root/src\",\"Entrypoint\":[\"/bin/env\"],\"OnBuild\":null,\"Labels\":{\"build-date\":\"20161214\",\"io.k8s.description\":\"Platform for building and running Ruby 2.2 applications\",\"io.k8s.display-name\":\"lgrpoj/ruby-ex-1:5ed595da\",\"io.openshift.build.commit.author\":\"Ionut Palade \\u003cPI-Victor.github.com\\u003e\",\"io.openshift.build.commit.date\":\"Mon Dec 12 14:37:32 2016 +0100\",\"io.openshift.build.commit.id\":\"855ab2de53ff897a19e1055f7554c64d19e02c50\",\"io.openshift.build.commit.message\":\"Merge pull request #6 from aj07/typo\",\"io.openshift.build.commit.ref\":\"master\",\"io.openshift.build.image\":\"centos/ruby-22-centos7@sha256:460730627b1c32c10368aa754e7075e447394b76c5a213a819a11297000edd2d\",\"io.openshift.build.source-location\":\"https://github.com/openshift/ruby-ex.git\",\"io.openshift.builder-base-version\":\"bfd4736\",\"io.openshift.builder-version\":\"06c0ec324aa6edf151f4ea1e7304199c72011bec\",\"io.openshift.expose-services\":\"8080:http\",\"io.openshift.s2i.scripts-url\":\"image:///usr/libexec/s2i\",\"io.openshift.tags\":\"builder,ruby,ruby22\",\"io.s2i.scripts-url\":\"image:///usr/libexec/s2i\",\"license\":\"GPLv2\",\"name\":\"CentOS Base Image\",\"vendor\":\"CentOS\"}},\"created\":\"2017-02-09T10:01:39.607932417Z\",\"docker_version\":\"1.10.3\",\"id\":\"2544e3a42489d71f7fb05328d8b5bcb7b0ed3f00d2418012d514a794b16e6f7b\",\"os\":\"linux\",\"parent\":\"62a557a72493406068264efccf4a03a6b15446d1a114af55f0756f997f60e7ec\"}"
      },
      {
         "v1Compatibility": "{\"id\":\"62a557a72493406068264efccf4a03a6b15446d1a114af55f0756f997f60e7ec\",\"parent\":\"5f61032c60977ebb9f35ae87289fdaed21de0c6f2cb2eaf9b803a116c051c79a\",\"created\":\"2017-02-07T16:23:34.396064385Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) LABEL io.openshift.builder-version=06c0ec324aa6edf151f4ea1e7304199c72011bec\"]},\"author\":\"SoftwareCollections.org \\u003csclorg\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"5f61032c60977ebb9f35ae87289fdaed21de0c6f2cb2eaf9b803a116c051c79a\",\"parent\":\"3f30f7308b8edb50f6fdd058bd7d9c504445538c94c02ede67f65a197941044c\",\"created\":\"2017-02-07T16:23:26.981260791Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) CMD [\\\"/bin/sh\\\" \\\"-c\\\" \\\"$STI_SCRIPTS_PATH/usage\\\"]\"]},\"author\":\"SoftwareCollections.org \\u003csclorg\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"3f30f7308b8edb50f6fdd058bd7d9c504445538c94c02ede67f65a197941044c\",\"parent\":\"f6361783d5bd16bafd68299087d6fb0e7fb9a66299f9267c998b9fc379e3af80\",\"created\":\"2017-02-07T16:23:19.402056901Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) USER [1001]\"]},\"author\":\"SoftwareCollections.org \\u003csclorg\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"f6361783d5bd16bafd68299087d6fb0e7fb9a66299f9267c998b9fc379e3af80\",\"parent\":\"721be1a2a1d46246b5f265f833744eb47532b3f9c198e7b623a7dcc6d12c5362\",\"created\":\"2017-02-07T16:23:09.117473874Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c chown -R 1001:0 /opt/app-root \\u0026\\u0026 chmod -R ug+rwx /opt/app-root\"]},\"author\":\"SoftwareCollections.org \\u003csclorg\\u003e\"}"
      },
      {
         "v1Compatibility": "{\"id\":\"721be1a2a1d46246b5f265f833744eb47532b3f9c198e7b623a7dcc6d12c5362\",\"parent\":\"a50f754e2c44ae39e62ab368a9230801c44fcf2b512b6db157432f467b97c339\",\"created\":\"2017-02-07T16:23:00.355368979Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) COPY dir:4abb8aa4037d4ccf9187e54dddd8fd759645a008de59c2ae03f09f6c510a90a9 in /opt/app-root\"]},\"author\":\"SoftwareCollections.org \\u003csclorg\\u003e\"}"
      },
      {
         "v1Compatibility": "{\"id\":\"a50f754e2c44ae39e62ab368a9230801c44fcf2b512b6db157432f467b97c339\",\"parent\":\"d53f1df78daf1e1fe406b75d0ea1cb128e1bbef1d9587a9aa8c85cf9f61c1a10\",\"created\":\"2017-02-07T16:22:47.236542373Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) COPY dir:b1da86c0f4ca04fed6e7bd8c3fa3c9896deba393695ed3c753a8a1a088f60c12 in /usr/libexec/s2i\"]},\"author\":\"SoftwareCollections.org \\u003csclorg\\u003e\"}"
      },
      {
         "v1Compatibility": "{\"id\":\"d53f1df78daf1e1fe406b75d0ea1cb128e1bbef1d9587a9aa8c85cf9f61c1a10\",\"parent\":\"118464f0bb32ab79be2bbb1d6a41319e939316f0bdcd6ab0705ede7cc9a951a6\",\"created\":\"2017-02-07T16:22:30.530473423Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c INSTALL_PKGS=\\\"rh-ruby22 rh-ruby22-ruby-devel rh-ruby22-rubygem-rake v8314 rh-ruby22-rubygem-bundler nodejs010\\\" \\u0026\\u0026     yum install -y centos-release-scl \\u0026\\u0026     yum install -y --setopt=tsflags=nodocs  $INSTALL_PKGS \\u0026\\u0026 rpm -V $INSTALL_PKGS \\u0026\\u0026     yum clean all -y\"]},\"author\":\"SoftwareCollections.org \\u003csclorg\\u003e\"}"
      },
      {
         "v1Compatibility": "{\"id\":\"118464f0bb32ab79be2bbb1d6a41319e939316f0bdcd6ab0705ede7cc9a951a6\",\"parent\":\"57f6f68f2c2c016843b9c1b794962413ae371c98d385e8fe3f69b4044184b6fa\",\"created\":\"2017-02-07T16:19:30.06429988Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) LABEL io.k8s.description=Platform for building and running Ruby 2.2 applications io.k8s.display-name=Ruby 2.2 io.openshift.expose-services=8080:http io.openshift.tags=builder,ruby,ruby22\"]},\"author\":\"SoftwareCollections.org \\u003csclorg\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"57f6f68f2c2c016843b9c1b794962413ae371c98d385e8fe3f69b4044184b6fa\",\"parent\":\"206c0879d9604c8dd2bf46750f00f4cfbd06986dcba5c556715e45562adce70f\",\"created\":\"2017-02-07T16:19:24.434230875Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) ENV RUBY_VERSION=2.2\"]},\"author\":\"SoftwareCollections.org \\u003csclorg\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"206c0879d9604c8dd2bf46750f00f4cfbd06986dcba5c556715e45562adce70f\",\"parent\":\"b00834916802b3d98f480b9cecaa0ffba9fb5591608574175fc07733485a2cfd\",\"created\":\"2017-02-07T16:19:18.827146476Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) EXPOSE 8080/tcp\"]},\"author\":\"SoftwareCollections.org \\u003csclorg\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"b00834916802b3d98f480b9cecaa0ffba9fb5591608574175fc07733485a2cfd\",\"parent\":\"6366dab18770823fb79819a2fbc42834ccdae91046366c539c5772ce334620ed\",\"created\":\"2017-02-07T16:19:11.947245564Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) MAINTAINER SoftwareCollections.org \\u003csclorg\\u003e\"]},\"author\":\"SoftwareCollections.org \\u003csclorg\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"6366dab18770823fb79819a2fbc42834ccdae91046366c539c5772ce334620ed\",\"parent\":\"88dec2b607ddf20dd46cc8839e59abafe25eba8b1df10e7e3e09a03143775ba6\",\"created\":\"2017-02-07T14:40:06.888994193Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) LABEL io.openshift.builder-base-version=bfd4736\"]},\"author\":\"Jakub Hadvig \\u003cjhadvig\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"88dec2b607ddf20dd46cc8839e59abafe25eba8b1df10e7e3e09a03143775ba6\",\"parent\":\"86e1c27719bdd532e3c37323e52ad78d47cb3aa6eea83aa382fa4ed99b6a4839\",\"created\":\"2017-02-07T14:40:02.194720807Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) CMD [\\\"base-usage\\\"]\"]},\"author\":\"Jakub Hadvig \\u003cjhadvig\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"86e1c27719bdd532e3c37323e52ad78d47cb3aa6eea83aa382fa4ed99b6a4839\",\"parent\":\"86bf6d084a80df2eb177d209a12a81022ca28ce2eb1963356e88b5fd8ea40698\",\"created\":\"2017-02-07T14:39:57.866183301Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) ENTRYPOINT \\u0026{[\\\"container-entrypoint\\\"]}\"]},\"author\":\"Jakub Hadvig \\u003cjhadvig\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"86bf6d084a80df2eb177d209a12a81022ca28ce2eb1963356e88b5fd8ea40698\",\"parent\":\"8d15ff5d1ef88afe37983b8e302294174e2de965b1abe6e326d928e5df6156b3\",\"created\":\"2017-02-07T14:39:51.919682393Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) WORKDIR /opt/app-root/src\"]},\"author\":\"Jakub Hadvig \\u003cjhadvig\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"8d15ff5d1ef88afe37983b8e302294174e2de965b1abe6e326d928e5df6156b3\",\"parent\":\"a3086aa2a7b909ad3f96307e7f400a5f1e1acec2c8ab425724659a518ac0a511\",\"created\":\"2017-02-07T14:39:48.171094843Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) COPY dir:7c2ced3187c8deac6a4ede38feda6eaacbef85bf4975d7979aea4b79db005ca8 in /usr/bin/\"]},\"author\":\"Jakub Hadvig \\u003cjhadvig\\u003e\"}"
      },
      {
         "v1Compatibility": "{\"id\":\"a3086aa2a7b909ad3f96307e7f400a5f1e1acec2c8ab425724659a518ac0a511\",\"parent\":\"4bf4f26f8082668cd81b45378622ef918ca73edf60e84ae542c09732656b55f3\",\"created\":\"2016-12-20T10:00:15.084189736Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c rpmkeys --import file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7 \\u0026\\u0026   INSTALL_PKGS=\\\"autoconf   automake   bsdtar   bzip2   findutils   gcc-c++   gd-devel   gdb   gettext   git   libcurl-devel   libxml2-devel   libxslt-devel   lsof   make   mariadb-devel   mariadb-libs   openssl-devel   patch   postgresql-devel   procps-ng   scl-utils   sqlite-devel   tar   unzip   wget   which   yum-utils   zlib-devel\\\" \\u0026\\u0026   mkdir -p ${HOME}/.pki/nssdb \\u0026\\u0026   chown -R 1001:0 ${HOME}/.pki \\u0026\\u0026   yum install -y --setopt=tsflags=nodocs $INSTALL_PKGS \\u0026\\u0026   rpm -V $INSTALL_PKGS \\u0026\\u0026   yum clean all -y \\u0026\\u0026   useradd -u 1001 -r -g 0 -d ${HOME} -s /sbin/nologin       -c \\\"Default Application User\\\" default \\u0026\\u0026   chown -R 1001:0 /opt/app-root\"]},\"author\":\"Jakub Hadvig \\u003cjhadvig\\u003e\"}"
      },
      {
         "v1Compatibility": "{\"id\":\"4bf4f26f8082668cd81b45378622ef918ca73edf60e84ae542c09732656b55f3\",\"parent\":\"6f2595bffdc30a81ae41b5e2244e9ec46a25951842349002da5186243253d17a\",\"created\":\"2016-12-20T09:57:33.884461265Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) ENV BASH_ENV=/opt/app-root/etc/scl_enable ENV=/opt/app-root/etc/scl_enable PROMPT_COMMAND=. /opt/app-root/etc/scl_enable\"]},\"author\":\"Jakub Hadvig \\u003cjhadvig\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"6f2595bffdc30a81ae41b5e2244e9ec46a25951842349002da5186243253d17a\",\"parent\":\"b7a9f751f2d64842b35f56eebd2f87929aa4e34d440cd551b394344972b418ba\",\"created\":\"2016-12-20T09:57:13.815581712Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) COPY file:ebe723f29ba98a751f6afca41fea401f2c20174f413ab2a7eb4eaeaa9a322cc7 in /opt/app-root/etc/scl_enable\"]},\"author\":\"Jakub Hadvig \\u003cjhadvig\\u003e\"}"
      },
      {
         "v1Compatibility": "{\"id\":\"b7a9f751f2d64842b35f56eebd2f87929aa4e34d440cd551b394344972b418ba\",\"parent\":\"e78b2fb0ee8d9af79e3dae18cc9702c087ca9de69ae071dbb74a27becf4bf07a\",\"created\":\"2016-12-20T09:57:01.090279875Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) ENV STI_SCRIPTS_URL=image:///usr/libexec/s2i STI_SCRIPTS_PATH=/usr/libexec/s2i HOME=/opt/app-root/src PATH=/opt/app-root/src/bin:/opt/app-root/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"]},\"author\":\"Jakub Hadvig \\u003cjhadvig\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"e78b2fb0ee8d9af79e3dae18cc9702c087ca9de69ae071dbb74a27becf4bf07a\",\"parent\":\"4ef929f1a369bd03c4c91e32b0e6d62c0fbdae92a7563aeb657b3102ade0f62a\",\"created\":\"2016-12-20T09:56:43.287284059Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) LABEL io.openshift.s2i.scripts-url=image:///usr/libexec/s2i io.s2i.scripts-url=image:///usr/libexec/s2i\"]},\"author\":\"Jakub Hadvig \\u003cjhadvig\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"4ef929f1a369bd03c4c91e32b0e6d62c0fbdae92a7563aeb657b3102ade0f62a\",\"parent\":\"08e1d80f2b80b86e13801c44c691fbc5812755972d44c1b2942f7f0cfa00a438\",\"created\":\"2016-12-20T09:56:30.945436702Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) MAINTAINER Jakub Hadvig \\u003cjhadvig\\u003e\"]},\"author\":\"Jakub Hadvig \\u003cjhadvig\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"08e1d80f2b80b86e13801c44c691fbc5812755972d44c1b2942f7f0cfa00a438\",\"parent\":\"a04895de19968b0665115154295009f75bcba85eb71dbb7fe6ae14cb8c95a475\",\"created\":\"2016-12-15T18:21:23.875991987Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop)  CMD [\\\"/bin/bash\\\"]\"]},\"author\":\"https://github.com/CentOS/sig-cloud-instance-images\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"a04895de19968b0665115154295009f75bcba85eb71dbb7fe6ae14cb8c95a475\",\"parent\":\"0a444b299d5a7868b640b12020a2684a72167b6710037b6bd4933f6713b2b47c\",\"created\":\"2016-12-15T18:21:23.00781987Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop)  LABEL name=CentOS Base Image vendor=CentOS license=GPLv2 build-date=20161214\"]},\"author\":\"https://github.com/CentOS/sig-cloud-instance-images\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"0a444b299d5a7868b640b12020a2684a72167b6710037b6bd4933f6713b2b47c\",\"parent\":\"3690474eb5b4b26fdfbd89c6e159e8cc376ca76ef48032a30fa6aafd56337880\",\"created\":\"2016-12-15T18:21:21.197722058Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) ADD file:940c77b6724c00d4208cc72169a63951eaa605672bcc5902ab2013cbae107434 in / \"]},\"author\":\"https://github.com/CentOS/sig-cloud-instance-images\"}"
      },
      {
         "v1Compatibility": "{\"id\":\"3690474eb5b4b26fdfbd89c6e159e8cc376ca76ef48032a30fa6aafd56337880\",\"created\":\"2016-08-30T18:18:45.109943798Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop)  MAINTAINER https://github.com/CentOS/sig-cloud-instance-images\"]},\"author\":\"https://github.com/CentOS/sig-cloud-instance-images\",\"throwaway\":true}"
      }
   ],
   "signatures": [
      {
         "header": {
            "jwk": {
               "crv": "P-256",
               "kid": "HWTR:AWP6:JIJF:VIQT:5RR2:DM7W:T3ED:47JU:D5C6:M7JY:5HQJ:C5CV",
               "kty": "EC",
               "x": "x45hKKUB7gB98oZJod5FA6W3mqF49kfiTol5F1Rn8WE",
               "y": "b9lpc4XHhhrPv1kfFXBvWj4WBVXh2N2DkIT3VIhhzvQ"
            },
            "alg": "ES256"
         },
         "signature": "NuAQnUGV4kjrepkj5bMCiJVctpFSenMw-OHDfIMNxzIs6nLFxwGjcVjBv4RUYvbMXxy_PtCS8_xmbF5OcaHofQ",
         "protected": "eyJmb3JtYXRMZW5ndGgiOjIwNzM5LCJmb3JtYXRUYWlsIjoiQ24wIiwidGltZSI6IjIwMTctMDItMDlUMTA6MDI6MjNaIn0"
      }
   ]

Comment 3 Michal Minar 2017-02-09 18:11:12 UTC
Thanks @geliu!

Migration script PR: https://github.com/openshift/ose/pull/608

Comment 6 ge liu 2017-02-17 09:49:37 UTC
@miminar I tried to run the test scenarios following your comment 5, but I blocked at some points:
1. Install some old 3.3 cluster (lacking PR#582) ==>could u help to provide fork ami name or detailed 3.3.x.xx version info to me?
6. Update to the latest 3.3 cluster (having PR#582) - don't forget to update the registry image as well.  ==>fork ami name and regarding to 'update the registry image', I think could do it by run command ‘oadm registry --latest-images=true’ to install the registry with latest image, right?

Comment 7 Michal Minar 2017-02-21 09:27:16 UTC
> could u help to provide fork ami name or detailed 3.3.x.xx version info to me?

The fork_ami with 3.3 without the manifest migration patch will be ready at: https://ci.openshift.redhat.com/jenkins/job/fork_ami/325/

> 6. Update to the latest 3.3 cluster (having PR#582) …  ==>fork ami name

https://ci.openshift.redhat.com/jenkins/job/fork_ami/321/

> and regarding to 'update the registry image', I think could do it by run command ‘oadm registry --latest-images=true’ to install the registry with latest image, right?

I'm not sure. I don't know how do you consume the images from fork_amis. I
don't expect the images to be tagged under ':latest' tag somewhere so I don't
think the `--latest-images=true` will work but I may be wrong. Please, reach
me on IRC if you have problems deploying a registry from 325 fork_ami.

Comment 8 Michal Minar 2017-02-21 14:07:55 UTC
The fork_ami build for 3.3 without the manifest migration patch mentioned in comment 7 has failed. Here's a correct link:
https://ci.openshift.redhat.com/jenkins/job/fork_ami/327/

Comment 9 Michal Minar 2017-02-22 16:02:46 UTC
The builds have been overriden by builds for 3.4 because they referenced the same trello card. Here's new fork_ami build for 3.3 lacking the fix: https://ci.openshift.redhat.com/jenkins/job/fork_ami/332/

Comment 10 Michal Minar 2017-02-22 16:10:54 UTC
And here's a new fork_ami for 3.3 with the fix applied:
https://ci.openshift.redhat.com/jenkins/job/fork_ami/333/

Comment 11 Michal Minar 2017-02-22 16:20:01 UTC
And the fork_ami without the manifest migration patch mentioned in comment 8 should still be valid: https://ci.openshift.redhat.com/jenkins/job/fork_ami/327/

Comment 12 ge liu 2017-02-23 10:55:21 UTC
@Miminar, regarding to comment 10, the fork-ami url is error, could you help to double check it? thx

###################################################################
And here's a new fork_ami for 3.3 with the fix applied:
https://ci.openshift.redhat.com/jenkins/job/fork_ami/333/
###################################################################

Comment 13 Michal Minar 2017-02-23 15:06:39 UTC
Yeah, sorry about that. For some reason (unknown to me), the builds have been completely deleted. I've just started new ones:

https://ci.openshift.redhat.com/jenkins/job/fork_ami/337/ with the fix not applied
https://ci.openshift.redhat.com/jenkins/job/fork_ami/338/ with the fix applied

I hope they survive until you finish testing.

Comment 14 Michal Minar 2017-02-23 21:43:49 UTC
The former [1] succeeded. The later failed though. If the flaking gods allow, the fork_ami with the fix applied will be at [2].

[1] https://ci.openshift.redhat.com/jenkins/job/fork_ami/337/
[2] https://ci.openshift.redhat.com/jenkins/job/fork_ami/341/

Comment 15 ge liu 2017-02-27 02:48:37 UTC
@Miminar, the fork_ami [1][2] is not exists now, could u help to rebuild it? thx

Comment 16 Michal Minar 2017-02-28 08:45:28 UTC
@geliu, I'm trying for the last couple of days. Dependencies fail to install [1]. I'll give it one more try and then call Steve K. for help.

[1] https://ci.openshift.redhat.com/jenkins/job/fork_ami/350/console

Comment 17 ge liu 2017-02-28 09:45:03 UTC
@miminar, I tested it in recently, and got the result today, there is some problem in ocp 3.3 based on my test result, the test steps listed below:

1). Install ocp 3.3.1.13 (build date:2017-02-07.1) that without fix and migrated script

2). Prepared data(sti build and import images)

3). Get the manifest from image:

    # oc get image -o  go-template=$'{{if .dockerImageManifest}}present{{else}}removed{{end}}\n' "sha256:ae8867c752f5f9208020593646d6658db2eb3cf7a15acbb16dedb4b049cc00d6"
present

# oc get image -o  go-template=$'{{if .dockerImageManifest}}present{{else}}removed{{end}}\n' "sha256:26c024386920b27731d5c1e615ab84a6a4bf623fc1f0a8307eab3d5da580ad84"
present


4). Install ocp 3.3.1.15(build date: 2017-02-24.1) that added fix and migrated script and save 2 stuff below:
    (1). brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/ose-docker-registry   v3.3.1.15   ==> docker push into docker.io/geliu/
    (2). binary file: /usr/bin/openshift


5). Go back to first ocp 3.3.1.13 env, replace /usr/bin/openshift with the one(version: 3.3.1.15) and replace the registry image with version: 3.3.1.15
, and restart openshift master service

6). run migrate script:

# ./migrate-image-manifests.sh -a -r http://172.30.xx.xx:5000  -t hwbwTksFMMZhp49gm6-2gZk_RIffPl-zbL6jo8N3-ZM 
Processing project 'default'
Processing project 'kube-system'
Processing project 'lgproj'
Migrating managed isimage 'lgproj/origin-ruby-sample@sha256:26c024386920b27731d5c1e615ab84a6a4bf623fc1f0a8307eab3d5da580ad84'
HTTP/1.1 200 OK
Migrating external isimage 'lgproj/ruby-22-centos7@sha256:9182cb1502d5967bbc727488586ee4b52da9d509f0d7d01bf14c3fb2faa39c43'
HTTP/1.1 200 OK
Migrating managed isimage 'lgproj/ruby-ex@sha256:ae8867c752f5f9208020593646d6658db2eb3cf7a15acbb16dedb4b049cc00d6'
HTTP/1.1 200 OK
Processing project 'logging'
Processing project 'management-infra'
Processing project 'openshift'
Migrating external isimage 'openshift/dotnet@sha256:bca54b9e42de51b8962cc8601416ecdedf3f25139c0ba5c9ed49cd27506fdc9b'


7). check the manifest in image, it still exist:

# oc get image -o  go-template=$'{{if .dockerImageManifest}}present{{else}}removed{{end}}\n' "sha256:ae8867c752f5f9208020593646d6658db2eb3cf7a15acbb16dedb4b049cc00d6"
present


8). oc get image sha:xxxx and found the manifest still exist

9). run curl command to check the manifest in registry, it exist also:

curl -u geliu:hwbwTksFMMZhp49gm6-2gZk_RIffPl-zbL6jo8N3-ZM  http://172.30.xxx.xxx:5000/v2/lgproj/origin-ruby-sample/manifests/sha256:26c024386920b27731d5c1e615ab84a6a4bf623fc1f0a8307eab3d5da580ad84
{
   "schemaVersion": 1,
   "name": "lgproj/origin-ruby-sample",
   "tag": "latest",
   "architecture": "amd64",
   "fsLayers": [
      {
         "blobSum": "sha256:129a8d27bb3231f7cb9d5b84c065f1b116774a35acedbedb430dea67b51d2ce4"
      },
...............


So the test conclustion is: migrate script move the manifest data from etcd to registry, and don't remove it in etcd, this is a problem.

Comment 18 Michal Minar 2017-02-28 11:00:38 UTC
I'm afraid there's one more thing you need to do before running the migration script. Cluster roles need to be reconciled to allow registry to do the image updates.

Please, try to run `oadm policy reconcile-cluster-roles --confirm` and then re-run the migration script.

Comment 19 ge liu 2017-03-01 01:59:46 UTC
@miminar, thanks for investigating again, a couple of issues:

1). The problem still exist after run 'oadm' command mentioned in comment 18, the menifest still exist in etcd
   # oadm policy reconcile-cluster-roles --confirm
      clusterrole/system:registry

2). Regrading to 'oadm' command, I really got some prompt msg when run migrate scripts, and I done it following the prompt msg, but there isn't prompt msg you mentioned in comment 18, so my question is: is these command line mandatory? if yes, the migrate script should prompt it to customer. I checked my test steps history, the below oadm commands are prompt when run migrate script:

  192  ./migrate-image-manifests.sh -r http://172.xx.xx.xx:5000  -t hwbwTksFMMZhp49gm6-2gZk_RIffPl-zbL6jo8N3-ZM
  193  oadm policy add-cluster-role-to-user system:image-auditor geliu
  198  oadm policy add-cluster-role-to-user registry-viewer  geliu

Comment 20 Michal Minar 2017-03-01 15:55:34 UTC
We recommend [1] running the `oadm policy reconcile-cluster-roles --confirm` during each cluster upgrade so I don't think it needs to be mentioned by the script.

[1] https://docs.openshift.org/latest/install_config/upgrading/manual_upgrades.html#updating-policy-definitions

Could you please provide me an access to the test machine? Last time I tried I didn't encounter a problem with deleting the images.

One more question, are the images that you tested with tagged in at least one imagestream? If not, the migration script won't deal with them. And image pruner will probably delete them.

Comment 21 ge liu 2017-03-02 05:02:53 UTC
@miminar, I agree with your comment about reconcile command, my test env was damaged yesterday, and my images was tagged in one imagestreams as my understanding, I create it with below commands:

oc import-image --confirm --from=centos/ruby-22-centos7 ruby-22-centos7:latest	
oc new-build --image-stream=ruby-22-centos7 https://github.com/openshift/ruby-ex.git

Comment 22 Michal Minar 2017-03-02 08:39:04 UTC
Glad to hear that, I'll re-create the fork_amis for you to re-test. Thanks.

Comment 23 Michal Minar 2017-03-07 09:05:26 UTC
@geliu it seems like the fork_amis
  https://ci.openshift.redhat.com/jenkins/job/fork_ami/337/
  https://ci.openshift.redhat.com/jenkins/job/fork_ami/352/

are ready for you to start testing. Can you please confirm?

Comment 24 Michal Minar 2017-03-07 09:25:55 UTC
This is a neverending story.
According to QE, the https://ci.openshift.redhat.com/jenkins/job/fork_ami/337/ is not available anymore.

I hope the 352 fork_ami will last until I manage to build a new one.

Comment 25 Michal Minar 2017-03-07 13:08:43 UTC
*** Bug 1378180 has been marked as a duplicate of this bug. ***

Comment 26 ge liu 2017-03-13 10:49:05 UTC
@miminar, I tested with your new fork_ami, the test result just like comment 17,
test conclustion is: migrate script move the manifest data from etcd to registry, and don't remove it in etcd, this is a problem.


old origin version:
openshift v1.3.0-alpha.3+5cb02f1
kubernetes v1.3.0+52492b4
etcd 2.3.0+git

new origin version:
openshift v1.3.0-alpha.3+003c820
kubernetes v1.3.0+52492b4
etcd 2.3.0+git


# ./migrate-image-manifests.sh -r 172.x.x.x:5000  -t up8e8H8B13WMcVHLo9hxs2HK6fwj6PSLjqawgEm8i3w -a
Processing project 'lgproj1'
Migrating managed isimage 'lgproj1/origin-ruby-sample@sha256:21ea2ba61d4ee45c0e9c007683200dd9313194ad86f8073b1562e5ed40af7176'
HTTP/1.1 200 OK
Migrating external isimage 'lgproj1/ruby-22-centos7@sha256:9182cb1502d5967bbc727488586ee4b52da9d509f0d7d01bf14c3fb2faa39c43'
HTTP/1.1 200 OK
# export KUBECONFIG=/root/openshift.local.config/master/admin.kubeconfig
#  oc get image -o go-template=$'{{if .dockerImageManifest}}present{{else}}removed{{end}}\n'  sha256:21ea2ba61d4ee45c0e9c007683200dd9313194ad86f8073b1562e5ed40af7176
present

# curl -u lg:up8e8H8B13WMcVHLo9hxs2HK6fwj6PSLjqawgEm8i3w http://172.30.x.x:5000/v2/lgproj1/origin-ruby-sample/manifests/latest
{
   "schemaVersion": 1,
   "name": "lgproj1/origin-ruby-sample",
   "tag": "latest",
   "architecture": "amd64",
   "fsLayers": [
      {
         "blobSum": "sha256:aa31f57bc506919f8c0923a8f274bc5c13d5bab848b3dafce6fb60a7a3bdeed3"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
...............

Comment 27 Michal Minar 2017-03-23 11:21:21 UTC
@geliu I've tried one more time. I found out that there are few more permissions necessary in order to migrate images tagged in all namespaces. With the recent version of the script, if the running user cannot list all the projects in the cluster, only projects he owns will be processed.

I've updated the script to check for additional permissions:

 https://raw.githubusercontent.com/miminar/ose/f7afb0695671ce6cf3edf59f313ed7fc27afd126/contrib/migration/migrate-image-manifests.sh?token=ABaQkwf3q-PJIsWnHWbH8A3Sji1GpYcWks5Y3NtawA%3D%3D

PR is at https://github.com/openshift/ose/pull/659.

Can you please try once more with the updated script?

Comment 28 ge liu 2017-03-24 06:04:07 UTC
@mimimar, Done the test based on comment 27, and test result listed below, the migrate script works well as expected, but it still could not remove data saved in etcd, since this issue exists for a while acrossed last several round of test, so I will open a new bug to trace it.

old:
# openshift version
openshift v1.3.0-alpha.3+5cb02f1
kubernetes v1.3.0+52492b4
etcd 2.3.0+git

new:
# openshift version
openshift v1.3.0-alpha.3+003c820
kubernetes v1.3.0+52492b4
etcd 2.3.0+git

# ./migrate-image-manifests.sh 
Error from server: User "geliu" cannot get services in project "default"
failed to get service ip of svc/docker-registry
curl: (6) Could not resolve host: http; Name or service not known
Please, provide endpoint of integrated docker registry.

# ./migrate-image-manifests.sh -r http://172.x.x.5:5000 -t gHU4e0ITJmKLgL0OrPpYD8z4n-EB3wYRXA0VTok85eY
The user isn't authorized to get images!
The user isn't authorized to list images!
The user isn't authorized to update images!
The user isn't authorized to get projects!
Ask your admin to give you permissions to work with images, e.g.:
  oadm policy add-cluster-role-to-user system:image-auditor geliu
  oadm policy add-cluster-role-to-user cluster-reader geliu

# ./migrate-image-manifests.sh -r http://172.x.x.5:5000 -t gHU4e0ITJmKLgL0OrPpYD8z4n-EB3wYRXA0VTok85eY
Processing project 'default'
Processing project 'kube-system'
Processing project 'lgproj'
Would migrate external isimage 'lgproj/ruby-22-centos7@sha256:2c49b89506ae4edaeccaeae37643b30ebbc68ca19026de3ecb79e6a01cc77615'
Would migrate managed isimage 'lgproj/ruby-ex@sha256:2e11090f85bd8e8454094f3b77f081d3735cbc73bfb312e1816bfc6c73802f37'
Processing project 'lgproj1'
Would migrate managed isimage 'lgproj1/origin-ruby-sample@sha256:8d263809f79f85e7b77aae8313eadf5d88626718e3e45ea323e6aac4058456fe'
Processing project 'openshift'
Processing project 'openshift-infra'

Would migrate 3 out of 3 images.

# ./migrate-image-manifests.sh -r http://172.x.x.5:5000 -t gHU4e0ITJmKLgL0OrPpYD8z4n-EB3wYRXA0VTok85eY -aProcessing project 'default'
Processing project 'kube-system'
Processing project 'lgproj'
Migrating external isimage 'lgproj/ruby-22-centos7@sha256:2c49b89506ae4edaeccaeae37643b30ebbc68ca19026de3ecb79e6a01cc77615'
HTTP/1.1 200 OK
Migrating managed isimage 'lgproj/ruby-ex@sha256:2e11090f85bd8e8454094f3b77f081d3735cbc73bfb312e1816bfc6c73802f37'
HTTP/1.1 200 OK
Processing project 'lgproj1'
Migrating managed isimage 'lgproj1/origin-ruby-sample@sha256:8d263809f79f85e7b77aae8313eadf5d88626718e3e45ea323e6aac4058456fe'
HTTP/1.1 200 OK
Processing project 'openshift'
Processing project 'openshift-infra'

#  oc get image -o go-template=$'{{if .dockerImageManifest}}present{{else}}removed{{end}}\n' sha256:2e11090f85bd8e8454094f3b77f081d3735cbc73bfb312e1816bfc6c73802f37
present
#  oc get image -o go-template=$'{{if .dockerImageManifest}}present{{else}}removed{{end}}\n' sha256:8d263809f79f85e7b77aae8313eadf5d88626718e3e45ea323e6aac4058456fe
present
# curl -u geliu:gHU4e0ITJmKLgL0OrPpYD8z4n-EB3wYRXA0VTok85eY http://172.x.x.5:5000/v2/lgproj/ruby-ex/manifests/latest
{
   "schemaVersion": 1,
   "name": "lgproj/ruby-ex",
   "tag": "latest",
   "architecture": "amd64",
   "fsLayers": [
      {
         "blobSum": "sha256:d2a4419651c1aa4a7cda39a0291f799c4352a0a813457ded70e2e01234375127"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
.....................

Comment 29 ge liu 2017-03-24 06:19:56 UTC
@miminar, I filed a new bug to trace migrate script function: the image data have not be removed from etcd after migration(https://bugzilla.redhat.com/show_bug.cgi?id=1435536), current bug carried much update in it, if there is not other new function added, we may close it and fix the 'remove' function in my new bug, thanks.

Comment 30 Michal Minar 2017-03-27 06:28:40 UTC
I would rather continue with this bug. If the removal doesn't work, it means that the bug hasn't been fixed. I don't think it's a question of just the migration script. Nevertheless the patch and the script works for me.

Are you sure you have deployed the newer registry from the fork_ami with the migration patch applied?
Do you have the cluster still running so I can take a look?

Comment 31 Michal Minar 2017-03-27 06:31:52 UTC
*** Bug 1435536 has been marked as a duplicate of this bug. ***

Comment 32 ge liu 2017-03-27 06:41:46 UTC
I saved the newer registry image from fork_ami(docker push to docker.io) and replaced the old registry image with newer one(docker pull from docker.io).
the fork_ami instance run in aws env, and it will be erased everyday, we could not keep the env longer, that's why I asked you to take looks sereral times in before test round.

Comment 34 ge liu 2017-03-29 07:06:43 UTC
Sure, @miminar, we have verified the bug with fork_ami on origin env. thanks!

Comment 35 Michal Minar 2017-03-29 13:29:50 UTC
Thanks for the patience!

Comment 37 errata-xmlrpc 2017-06-15 18:36:44 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1425


Note You need to log in before you can comment on or make changes to this bug.