Bug 1418359 - [3.4] Image data stored in ETCD accounting for large DB.
Summary: [3.4] Image data stored in ETCD accounting for large DB.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image Registry
Version: 3.4.1
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 3.4.z
Assignee: Michal Minar
QA Contact: ge liu
URL:
Whiteboard:
Depends On: 1378180
Blocks: 1267746 1418358
TreeView+ depends on / blocked
 
Reported: 2017-02-01 16:07 UTC by Michal Minar
Modified: 2021-08-30 13:09 UTC (History)
20 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: OpenShift cluster used to store manifests of all images in etcd database. Consequence: The manifests occupied a lot of space in the database making it big and slow. Fix: The integrated registry now stores manifests in its associated storage rather than in etcd. Also manifests of remote images are not stored at all. They are fetch from external registries when needed. A migration script has been provided to move manifests from all existing images in the cluster into the integrated registry. Result: Newly pushed images will not cause etcd database to grow so fast. By using the migration script, the admin is able to reduce etcd size considerably.
Clone Of: 1378180
Environment:
Last Closed: 2017-03-15 20:02:17 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Verified test results (10.83 KB, text/plain)
2017-02-23 10:48 UTC, ge liu
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Origin (Github) 11925 0 None None None 2017-02-01 16:07:31 UTC
Red Hat Knowledge Base (Solution) 2688911 0 None None None 2017-02-01 16:07:31 UTC
Red Hat Product Errata RHBA-2017:0512 0 normal SHIPPED_LIVE OpenShift Container Platform 3.4.1.10, 3.3.1.17, and 3.2.1.28 bug fix update 2017-03-16 00:01:17 UTC

Comment 2 ge liu 2017-02-04 05:19:45 UTC
Run the test on fork ami env:

openshift v1.4.0-rc1+ed836c1-1103
kubernetes v1.4.0+776c994
etcd 3.1.0-rc.0

1.Prepareing image and is:
# oc import-image --confirm --from=centos/ruby-22-centos7 ruby-22-centos7:latest
# oc new-build --image-stream=ruby-22-centos7 https://github.com/openshift/ruby-ex.git

2. Check the image, make sure no manifest stored with the images:
# oc get images -o jsonpath=$'{.dockerImageManifest}\n' sha256:390712289d4d26663bf61f6680d9d2833bc9c2da5c4739f42cfb3e049de05628

[root@ip-172-xx-xx-xx xx]

3. Check the internal registry has stored the manifest:

# curl -u geliu:eNsb7YL4Y1xB30_LSqqGSvvs4bMiF6ouNzf3WvxPQ1M http://172.xx.xx.xx:5000/v2/lgproj/ruby-ex/manifests/latest
{
   "schemaVersion": 1,
   "name": "lgproj/ruby-ex",
   "tag": "latest",
   "architecture": "amd64",
   "fsLayers": [
      {
         "blobSum": "sha256:83ed987f155f37fc6b1857fec2ab913f525dc93a68db8680e588acf6ba4c3c39"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:0de12915e8b0f806de69a9e3e2477e2aa696f0be0bb6b04f7a799a7fd7d0efad"
      },
      {
         "blobSum": "sha256:5a96d3f5f6bed842da91866dcc2557566c9127bbfb38609b6de5d1e6305c0d6a"
      },
      {
         "blobSum": "sha256:c51e4ed5672f9fa8d90c422d14314d249438a7c737157b3e529ae5dbb3595630"
      },
      {
         "blobSum": "sha256:3d3aeee3d07bdafd4b2d837194567e92be864893c9a37f671fe0556c72d4a435"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a8870c236ab24c328ce1947923a36c4b950f99c88a377d33afea606279d6d287"
      },
      {
         "blobSum": "sha256:7199f1661dba67c2bbf4047b08ad638cb66d6666f4a540aec289fd5131adbcfe"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:088ea866df45bf386888938d2cc1cc19d4bfdbae9bdae23540d6a60ab2260487"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:17385548ba54dd00b2b73dc106749019fcc0cc5641317c97766f14647d06ad6e"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      }
   ],
   "history": [
      {
         "v1Compatibility": "{\"architecture\":\"amd64\",\"config\":{\"Hostname\":\"\",\"Domainname\":\"\",\"User\":\"1001\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"ExposedPorts\":{\"8080/tcp\":{}},\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"OPENSHIFT_BUILD_NAME=ruby-ex-1\",\"OPENSHIFT_BUILD_NAMESPACE=lgproj\",\"OPENSHIFT_BUILD_SOURCE=https://github.com/openshift/ruby-ex.git\",\"PATH=/opt/app-root/src/bin:/opt/app-root/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\",\"STI_SCRIPTS_URL=image:///usr/libexec/s2i\",\"STI_SCRIPTS_PATH=/usr/libexec/s2i\",\"HOME=/opt/app-root/src\",\"BASH_ENV=/opt/app-root/etc/scl_enable\",\"ENV=/opt/app-root/etc/scl_enable\",\"PROMPT_COMMAND=. /opt/app-root/etc/scl_enable\",\"RUBY_VERSION=2.2\"],\"Cmd\":[\"/usr/libexec/s2i/run\"],\"Image\":\"\",\"Volumes\":null,\"WorkingDir\":\"/opt/app-root/src\",\"Entrypoint\":[\"container-entrypoint\"],\"OnBuild\":null,\"Labels\":{\"build-date\":\"20161214\",\"io.k8s.description\":\"Platform for building and running Ruby 2.2 applications\",\"io.k8s.display-name\":\"lgproj/ruby-ex-1:d62f77b6\",\"io.openshift.build.commit.author\":\"Ionut Palade \\u003cPI-Victor.github.com\\u003e\",\"io.openshift.build.commit.date\":\"Mon Dec 12 14:37:32 2016 +0100\",\"io.openshift.build.commit.id\":\"855ab2de53ff897a19e1055f7554c64d19e02c50\",\"io.openshift.build.commit.message\":\"Merge pull request #6 from aj07/typo\",\"io.openshift.build.commit.ref\":\"master\",\"io.openshift.build.image\":\"centos/ruby-22-centos7@sha256:520dd8ee3d729f776318135e8f56ce7d36a08cb460a8906b2bb3db8cd2c97de0\",\"io.openshift.build.source-location\":\"https://github.com/openshift/ruby-ex.git\",\"io.openshift.builder-base-version\":\"a8deee2\",\"io.openshift.builder-version\":\"06c0ec324aa6edf151f4ea1e7304199c72011bec\",\"io.openshift.expose-services\":\"8080:http\",\"io.openshift.s2i.scripts-url\":\"image:///usr/libexec/s2i\",\"io.openshift.tags\":\"builder,ruby,ruby22\",\"io.s2i.scripts-url\":\"image:///usr/libexec/s2i\",\"license\":\"GPLv2\",\"name\":\"CentOS Base Image\",\"vendor\":\"CentOS\"}},\"container\":\"73e43a2fbcc848736de40591eadf25cf4b279eea24ae6fc53ec7f8faad2a4a6a\",\"container_config\":{\"Hostname\":\"ruby-ex-1-build\",\"Domainname\":\"\",\"User\":\"1001\",\"AttachStdin\":false,\"AttachStdout\":true,\"AttachStderr\":false,\"ExposedPorts\":{\"8080/tcp\":{}},\"Tty\":false,\"OpenStdin\":true,\"StdinOnce\":true,\"Env\":[\"OPENSHIFT_BUILD_NAME=ruby-ex-1\",\"OPENSHIFT_BUILD_NAMESPACE=lgproj\",\"OPENSHIFT_BUILD_SOURCE=https://github.com/openshift/ruby-ex.git\",\"PATH=/opt/app-root/src/bin:/opt/app-root/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\",\"STI_SCRIPTS_URL=image:///usr/libexec/s2i\",\"STI_SCRIPTS_PATH=/usr/libexec/s2i\",\"HOME=/opt/app-root/src\",\"BASH_ENV=/opt/app-root/etc/scl_enable\",\"ENV=/opt/app-root/etc/scl_enable\",\"PROMPT_COMMAND=. /opt/app-root/etc/scl_enable\",\"RUBY_VERSION=2.2\"],\"Cmd\":[\"/bin/sh\",\"-c\",\"tar -C /tmp -xf - \\u0026\\u0026 /usr/libexec/s2i/assemble\"],\"Image\":\"centos/ruby-22-centos7@sha256:520dd8ee3d729f776318135e8f56ce7d36a08cb460a8906b2bb3db8cd2c97de0\",\"Volumes\":null,\"WorkingDir\":\"/opt/app-root/src\",\"Entrypoint\":[\"container-entrypoint\"],\"OnBuild\":null,\"Labels\":{\"build-date\":\"20161214\",\"io.k8s.description\":\"Platform for building and running Ruby 2.2 applications\",\"io.k8s.display-name\":\"lgproj/ruby-ex-1:d62f77b6\",\"io.openshift.build.commit.author\":\"Ionut Palade \\u003cPI-Victor.github.com\\u003e\",\"io.openshift.build.commit.date\":\"Mon Dec 12 14:37:32 2016 +0100\",\"io.openshift.build.commit.id\":\"855ab2de53ff897a19e1055f7554c64d19e02c50\",\"io.openshift.build.commit.message\":\"Merge pull request #6 from aj07/typo\",\"io.openshift.build.commit.ref\":\"master\",\"io.openshift.build.image\":\"centos/ruby-22-centos7@sha256:520dd8ee3d729f776318135e8f56ce7d36a08cb460a8906b2bb3db8cd2c97de0\",\"io.openshift.build.source-location\":\"https://github.com/openshift/ruby-ex.git\",\"io.openshift.builder-base-version\":\"a8deee2\",\"io.openshift.builder-version\":\"06c0ec324aa6edf151f4ea1e7304199c72011bec\",\"io.openshift.expose-services\":\"8080:http\",\"io.openshift.s2i.scripts-url\":\"image:///usr/libexec/s2i\",\"io.openshift.tags\":\"builder,ruby,ruby22\",\"io.s2i.scripts-url\":\"image:///usr/libexec/s2i\",\"license\":\"GPLv2\",\"name\":\"CentOS Base Image\",\"vendor\":\"CentOS\"}},\"created\":\"2017-02-04T04:48:32.622773223Z\",\"docker_version\":\"1.12.5\",\"id\":\"00d987e06192cc770d7f92616548ead42b3a5d405b9b80fd01ee236d6550af56\",\"os\":\"linux\",\"parent\":\"81b8bb056c5dd1369752fbd7abf676ab04fb2992f48a579e54dcb44d8f58f868\"}"
      },
      {
         "v1Compatibility": "{\"id\":\"81b8bb056c5dd1369752fbd7abf676ab04fb2992f48a579e54dcb44d8f58f868\",\"parent\":\"706e1b919182a8bd9ef182e5264fbbb59cef43a46d0007fadf8c757f1ebd2b16\",\"created\":\"2017-02-01T13:34:00.747722256Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop)  LABEL io.openshift.builder-version=06c0ec324aa6edf151f4ea1e7304199c72011bec\"]},\"author\":\"SoftwareCollections.org \\u003csclorg\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"706e1b919182a8bd9ef182e5264fbbb59cef43a46d0007fadf8c757f1ebd2b16\",\"parent\":\"b0bfd8507b20dc0555a6eb3005d3dc3095aae65e676d8e7694e3b879d1bdc738\",\"created\":\"2017-02-01T13:33:58.159677945Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop)  CMD [\\\"/bin/sh\\\" \\\"-c\\\" \\\"$STI_SCRIPTS_PATH/usage\\\"]\"]},\"author\":\"SoftwareCollections.org \\u003csclorg\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"b0bfd8507b20dc0555a6eb3005d3dc3095aae65e676d8e7694e3b879d1bdc738\",\"parent\":\"fb908b872b5d6efb76e2d39df03e50066dd5b1d2bd7087fd1e8638bb29cf8828\",\"created\":\"2017-02-01T13:33:55.628701773Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop)  USER [1001]\"]},\"author\":\"SoftwareCollections.org \\u003csclorg\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"fb908b872b5d6efb76e2d39df03e50066dd5b1d2bd7087fd1e8638bb29cf8828\",\"parent\":\"23ca8b97c92bd123d1793ba18465ed600c2f6de44e94fb257439397470beaea4\",\"created\":\"2017-02-01T13:33:53.167492358Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c chown -R 1001:0 /opt/app-root \\u0026\\u0026 chmod -R ug+rwx /opt/app-root\"]},\"author\":\"SoftwareCollections.org \\u003csclorg\\u003e\"}"
      },
      {
         "v1Compatibility": "{\"id\":\"23ca8b97c92bd123d1793ba18465ed600c2f6de44e94fb257439397470beaea4\",\"parent\":\"6e0090788072a24b62933abf44da418681ee8a48f1a008ed36922cede899c788\",\"created\":\"2017-02-01T13:33:50.208909122Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) COPY dir:c8b38f11079d220fd53856c177b5c6c01ad8c44d056f8499d0ce839978a50213 in /opt/app-root \"]},\"author\":\"SoftwareCollections.org \\u003csclorg\\u003e\"}"
      },
      {
         "v1Compatibility": "{\"id\":\"6e0090788072a24b62933abf44da418681ee8a48f1a008ed36922cede899c788\",\"parent\":\"a800189a302f3ca478616ce2e53f46e367ab37d71a883a633f606afaa936d28f\",\"created\":\"2017-02-01T13:33:47.798917676Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) COPY dir:d367131025bc654a0234bfcc944c6e51b051d3c67febc95bb2ba93dc79763288 in /usr/libexec/s2i \"]},\"author\":\"SoftwareCollections.org \\u003csclorg\\u003e\"}"
      },
      {
         "v1Compatibility": "{\"id\":\"a800189a302f3ca478616ce2e53f46e367ab37d71a883a633f606afaa936d28f\",\"parent\":\"c72ea5466e690de3d139c248db8dded63df5afbbf3881869258b11ac769b8a9e\",\"created\":\"2017-02-01T13:33:45.143748912Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c INSTALL_PKGS=\\\"rh-ruby22 rh-ruby22-ruby-devel rh-ruby22-rubygem-rake v8314 rh-ruby22-rubygem-bundler nodejs010\\\" \\u0026\\u0026     yum install -y centos-release-scl \\u0026\\u0026     yum install -y --setopt=tsflags=nodocs  $INSTALL_PKGS \\u0026\\u0026 rpm -V $INSTALL_PKGS \\u0026\\u0026     yum clean all -y\"]},\"author\":\"SoftwareCollections.org \\u003csclorg\\u003e\"}"
      },
      {
         "v1Compatibility": "{\"id\":\"c72ea5466e690de3d139c248db8dded63df5afbbf3881869258b11ac769b8a9e\",\"parent\":\"5db83d9e55810434619968eb772701f3b94509e8ea30bf2c6d06c4b41e38b5e8\",\"created\":\"2017-02-01T13:33:03.734701543Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop)  LABEL io.k8s.description=Platform for building and running Ruby 2.2 applications io.k8s.display-name=Ruby 2.2 io.openshift.expose-services=8080:http io.openshift.tags=builder,ruby,ruby22\"]},\"author\":\"SoftwareCollections.org \\u003csclorg\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"5db83d9e55810434619968eb772701f3b94509e8ea30bf2c6d06c4b41e38b5e8\",\"parent\":\"8e526e95402c778af2796d0f3e932c78467f2e29436af33251106ad74dabd096\",\"created\":\"2017-02-01T13:33:01.489779935Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop)  ENV RUBY_VERSION=2.2\"]},\"author\":\"SoftwareCollections.org \\u003csclorg\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"8e526e95402c778af2796d0f3e932c78467f2e29436af33251106ad74dabd096\",\"parent\":\"92446529f14577fcb699405cf1ebc21afe18e7f352f30dd61202a55097482e10\",\"created\":\"2017-02-01T12:43:12.220688181Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop)  EXPOSE 8080/tcp\"]},\"author\":\"SoftwareCollections.org \\u003csclorg\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"92446529f14577fcb699405cf1ebc21afe18e7f352f30dd61202a55097482e10\",\"parent\":\"e9d2e7c99a49066424257186a15b5f3cb217d666fb0ff9941d17b3c0406e7e69\",\"created\":\"2017-02-01T12:43:09.682301382Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop)  MAINTAINER SoftwareCollections.org \\u003csclorg\\u003e\"]},\"author\":\"SoftwareCollections.org \\u003csclorg\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"e9d2e7c99a49066424257186a15b5f3cb217d666fb0ff9941d17b3c0406e7e69\",\"parent\":\"8614b8906c1c79658cfdf4f1bc4325c7779d404b65eab86c9df18a79a3036b9d\",\"created\":\"2016-12-20T10:00:42.599267404Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) LABEL io.openshift.builder-base-version=a8deee2\"]},\"author\":\"Jakub Hadvig \\u003cjhadvig\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"8614b8906c1c79658cfdf4f1bc4325c7779d404b65eab86c9df18a79a3036b9d\",\"parent\":\"23920117f35e661345ec3fbf37c61f2315617f1780e31a067a2949bae56b1170\",\"created\":\"2016-12-20T10:00:35.710198534Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) CMD [\\\"base-usage\\\"]\"]},\"author\":\"Jakub Hadvig \\u003cjhadvig\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"23920117f35e661345ec3fbf37c61f2315617f1780e31a067a2949bae56b1170\",\"parent\":\"ff9a10f6e370155173049053e243eb04d71af6566a319df2a8a7e521adfe4121\",\"created\":\"2016-12-20T10:00:30.853405402Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) ENTRYPOINT \\u0026{[\\\"container-entrypoint\\\"]}\"]},\"author\":\"Jakub Hadvig \\u003cjhadvig\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"ff9a10f6e370155173049053e243eb04d71af6566a319df2a8a7e521adfe4121\",\"parent\":\"50669dc72b57cfa5763f8d49a3113728f213a5d2f94ca0d226f387b60993ac42\",\"created\":\"2016-12-20T10:00:24.483187299Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) WORKDIR /opt/app-root/src\"]},\"author\":\"Jakub Hadvig \\u003cjhadvig\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"50669dc72b57cfa5763f8d49a3113728f213a5d2f94ca0d226f387b60993ac42\",\"parent\":\"e010aa9dbf88b582cd90c49ca3e55f5b22c7a9643804044d9dca8fc353c6b4d1\",\"created\":\"2016-12-20T10:00:18.951779558Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) COPY dir:47fe09828283c2e7fb1be367586884c397eff0e11002a92a95718cf3baa1ca11 in /usr/bin/\"]},\"author\":\"Jakub Hadvig \\u003cjhadvig\\u003e\"}"
      },
      {
         "v1Compatibility": "{\"id\":\"e010aa9dbf88b582cd90c49ca3e55f5b22c7a9643804044d9dca8fc353c6b4d1\",\"parent\":\"4a546c6e362f2512a54b3534bc80a7f72dce2c7ff83758a0f678331670ce01a0\",\"created\":\"2016-12-20T10:00:15.084189736Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c rpmkeys --import file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7 \\u0026\\u0026   INSTALL_PKGS=\\\"autoconf   automake   bsdtar   bzip2   findutils   gcc-c++   gd-devel   gdb   gettext   git   libcurl-devel   libxml2-devel   libxslt-devel   lsof   make   mariadb-devel   mariadb-libs   openssl-devel   patch   postgresql-devel   procps-ng   scl-utils   sqlite-devel   tar   unzip   wget   which   yum-utils   zlib-devel\\\" \\u0026\\u0026   mkdir -p ${HOME}/.pki/nssdb \\u0026\\u0026   chown -R 1001:0 ${HOME}/.pki \\u0026\\u0026   yum install -y --setopt=tsflags=nodocs $INSTALL_PKGS \\u0026\\u0026   rpm -V $INSTALL_PKGS \\u0026\\u0026   yum clean all -y \\u0026\\u0026   useradd -u 1001 -r -g 0 -d ${HOME} -s /sbin/nologin       -c \\\"Default Application User\\\" default \\u0026\\u0026   chown -R 1001:0 /opt/app-root\"]},\"author\":\"Jakub Hadvig \\u003cjhadvig\\u003e\"}"
      },
      {
         "v1Compatibility": "{\"id\":\"4a546c6e362f2512a54b3534bc80a7f72dce2c7ff83758a0f678331670ce01a0\",\"parent\":\"d77c77990212992106baacfe7b275707977f660f1ccf2d857dc3093057e9f404\",\"created\":\"2016-12-20T09:57:33.884461265Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) ENV BASH_ENV=/opt/app-root/etc/scl_enable ENV=/opt/app-root/etc/scl_enable PROMPT_COMMAND=. /opt/app-root/etc/scl_enable\"]},\"author\":\"Jakub Hadvig \\u003cjhadvig\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"d77c77990212992106baacfe7b275707977f660f1ccf2d857dc3093057e9f404\",\"parent\":\"c93211af9654da4e381881b0f9cb12366271f26880329f43b8d3f20f13318503\",\"created\":\"2016-12-20T09:57:13.815581712Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) COPY file:ebe723f29ba98a751f6afca41fea401f2c20174f413ab2a7eb4eaeaa9a322cc7 in /opt/app-root/etc/scl_enable\"]},\"author\":\"Jakub Hadvig \\u003cjhadvig\\u003e\"}"
      },
      {
         "v1Compatibility": "{\"id\":\"c93211af9654da4e381881b0f9cb12366271f26880329f43b8d3f20f13318503\",\"parent\":\"574bcfcba203a8f3e3f81acb62e36ff35df09d1db804019a6d93763e0617d8b4\",\"created\":\"2016-12-20T09:57:01.090279875Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) ENV STI_SCRIPTS_URL=image:///usr/libexec/s2i STI_SCRIPTS_PATH=/usr/libexec/s2i HOME=/opt/app-root/src PATH=/opt/app-root/src/bin:/opt/app-root/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"]},\"author\":\"Jakub Hadvig \\u003cjhadvig\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"574bcfcba203a8f3e3f81acb62e36ff35df09d1db804019a6d93763e0617d8b4\",\"parent\":\"eff4e3e9a3f521596c8981107eba5c48d54302655276148694f4455fef19bb3e\",\"created\":\"2016-12-20T09:56:43.287284059Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) LABEL io.openshift.s2i.scripts-url=image:///usr/libexec/s2i io.s2i.scripts-url=image:///usr/libexec/s2i\"]},\"author\":\"Jakub Hadvig \\u003cjhadvig\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"eff4e3e9a3f521596c8981107eba5c48d54302655276148694f4455fef19bb3e\",\"parent\":\"246a0cda8f6180e8e1a3be89374a0e0775ea16bbb8879cdc42f5aa18b20037f4\",\"created\":\"2016-12-20T09:56:30.945436702Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) MAINTAINER Jakub Hadvig \\u003cjhadvig\\u003e\"]},\"author\":\"Jakub Hadvig \\u003cjhadvig\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"246a0cda8f6180e8e1a3be89374a0e0775ea16bbb8879cdc42f5aa18b20037f4\",\"parent\":\"cb3ede6d84e8e3ff94d51e1edcaec9aee86e9daf9cf877df3f87aae1da0cac66\",\"created\":\"2016-12-15T18:21:23.875991987Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop)  CMD [\\\"/bin/bash\\\"]\"]},\"author\":\"https://github.com/CentOS/sig-cloud-instance-images\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"cb3ede6d84e8e3ff94d51e1edcaec9aee86e9daf9cf877df3f87aae1da0cac66\",\"parent\":\"b863b225c1bb592752ae04eb5295903d0cd97bb25621a653c7e38837b9e50009\",\"created\":\"2016-12-15T18:21:23.00781987Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop)  LABEL name=CentOS Base Image vendor=CentOS license=GPLv2 build-date=20161214\"]},\"author\":\"https://github.com/CentOS/sig-cloud-instance-images\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"b863b225c1bb592752ae04eb5295903d0cd97bb25621a653c7e38837b9e50009\",\"parent\":\"3690474eb5b4b26fdfbd89c6e159e8cc376ca76ef48032a30fa6aafd56337880\",\"created\":\"2016-12-15T18:21:21.197722058Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) ADD file:940c77b6724c00d4208cc72169a63951eaa605672bcc5902ab2013cbae107434 in / \"]},\"author\":\"https://github.com/CentOS/sig-cloud-instance-images\"}"
      },
      {
         "v1Compatibility": "{\"id\":\"3690474eb5b4b26fdfbd89c6e159e8cc376ca76ef48032a30fa6aafd56337880\",\"created\":\"2016-08-30T18:18:45.109943798Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop)  MAINTAINER https://github.com/CentOS/sig-cloud-instance-images\"]},\"author\":\"https://github.com/CentOS/sig-cloud-instance-images\",\"throwaway\":true}"
      }
   ],
   "signatures": [
      {
         "header": {
            "jwk": {
               "crv": "P-256",
               "kid": "REGM:UQN4:W5BH:X3JX:7O6R:YRDJ:2S55:QKFC:TCBB:UIJT:T33V:XMAF",
               "kty": "EC",
               "x": "iw2p5eG9u-hqr6t0biTuF20teRNCtAX5ZFuBPwyGc5s",
               "y": "LfKadEfgVX8WMM72Mw3xK8M_vlzCBRKfrh-n4N6y7Fw"
            },
            "alg": "ES256"
         },
         "signature": "v3phpwo3blXT3Q7aqp_GTR6HAm4kuZog2whW9fJVip-a9gU5fXegxtMG8pLv6MWJs9V2UDg_ITrtUn9zoC4KMw",
         "protected": "eyJmb3JtYXRMZW5ndGgiOjIwNzYxLCJmb3JtYXRUYWlsIjoiQ24wIiwidGltZSI6IjIwMTctMDItMDRUMDU6MDY6MTdaIn0"
      }
   ]

Comment 4 Michal Minar 2017-02-08 15:15:45 UTC
PR https://github.com/openshift/ose/pull/582 [merge]d for 3.4.

Comment 5 Michal Minar 2017-02-09 18:14:50 UTC
Migration script PR: https://github.com/openshift/ose/pull/607

Comment 7 ge liu 2017-02-17 09:56:43 UTC
@miminar, I need more info about the 3.4 fork ami or detailed version info that(lacking PR#582), and also the fork ami(having PR#582), thanks in advance.

Comment 8 Michal Minar 2017-02-21 09:46:47 UTC
The fork_ami with 3.4 branch without the migration patch (#582) will be hopefully ready at:
  https://ci.openshift.redhat.com/jenkins/job/fork_ami/326/

The 3.4 fork_ami with #582 is at:

  https://ci.openshift.redhat.com/jenkins/job/fork_ami/315/parameters/

Comment 9 Michal Minar 2017-02-21 16:16:20 UTC
Slight correction. The fork_ami with 3.4 without the migration patch is at https://ci.openshift.redhat.com/jenkins/job/fork_ami/328/

The fork_ami referenced in comment 8 failed unfortunately.

@geliu, please let me know if I can help in any way.

Comment 10 ge liu 2017-02-22 11:01:51 UTC
@miminar, I done step 1-5 in the suggested scrnario, but regarding to "step 6: Update to the latest 3.4 cluster (having PR#582) - don't forget to update the registry image as well."

my test execution plan is: start openshift cluster with form-aim(having PR#582), then copy openshift binary to replace the openshift cluster(lacking PR#582), then restart openshift service.

the openshift binary is: # which openshift
/data/src/github.com/openshift/origin/_output/local/bin/linux/amd64/openshift/*

do you think it's ok for the update operation? and currently issue is: there is not fork_ami with: (The 3.4 fork_ami with #582 is at:
https://ci.openshift.redhat.com/jenkins/job/fork_ami/315/parameters/), could u help to build a form_ami for me? thanks!

Comment 11 ge liu 2017-02-22 11:06:25 UTC
Suggested scrnarios:
Step 1-5:

1. swith to project/user: lgproj2/geliu, and do some builds:

#oc process -f https://raw.githubusercontent.com/openshift/origin/master/examples/sample-app/application-template-stibuild.json |oc create -f -

2.swith to project/user: proj-lg/lg, and import images

#oc import-image --confirm --from=centos/ruby-22-centos7 ruby-22-centos7:latest	
#oc new-build --image-stream=ruby-22-centos7 https://github.com/openshift/ruby-ex.git

3. check the images: dockerImageManifest

# oc get image -o go-template=$'{{if.dockerImageManifest}}present{{else}}removed{{end}}\n' sha256:8426f95b35ff6484360565155b470903bcffc82f0d87d72a0a528bcea9d8e0b8
present
# oc get image -o go-template=$'{{if.dockerImageManifest}}present{{else}}removed{{end}}\n' sha256:06f0fcff7cfbbf7aacd81d84efb0ce4d3c84be37bebbd35513f8f1fadab4b9a2 
present

it should works as expected in 3.4 cluster (lacking PR#582), and the next step is update to cluster (having PR#582)

Comment 12 Michal Minar 2017-02-22 11:52:02 UTC
>and currently issue is: there is not fork_ami with: (The 3.4 fork_ami with #582 is at: https://ci.openshift.redhat.com/jenkins/job/fork_ami/315/parameters/), could u help to build a form_ami for me? thanks!

@geliu I don't quite understand. You should be able to pull the fork_ami build from https://ci.openshift.redhat.com/jenkins/job/fork_ami/315/ the same way as from https://ci.openshift.redhat.com/jenkins/job/fork_ami/328/.

How do you check whether there's a build or not?

> do you think it's ok for the update operation? 

The registry needs to be re-deployed as well. Otherwise it looks good to me.

Comment 13 Wang Haoran 2017-02-22 15:31:01 UTC
As we talked from IRC, Michal will prepare two ami, and we will test like this:
1. Create all-in-one instance using the ami that don't have bug fix
2. prepare data
3. launch instance using the new ami with bug fix, and copy the openshift binary to old instance, also the registry image.(we can push the image to a external registry, then pull it in the old instance)
4. start the cluster in old instance using new binary, modify the dc/docker-registry to using the new registry image
.....

Comment 14 Michal Minar 2017-02-22 15:53:38 UTC
Ok, I've just started a new fork_ami build at: https://ci.openshift.redhat.com/jenkins/job/fork_ami/331/
Let's see if it passes.

Comment 15 Michal Minar 2017-02-22 21:01:48 UTC
For some reason, the build from comment 14 has been deleted. I've started a new one at https://ci.openshift.redhat.com/jenkins/job/fork_ami/335/. It has the #582 fix applied.

The second fork_ami with the fix not applied is at https://ci.openshift.redhat.com/jenkins/job/fork_ami/328/.

Comment 16 ge liu 2017-02-23 10:48:57 UTC
Created attachment 1256863 [details]
Verified test results

Verified and test steps in attached file.

Comment 21 errata-xmlrpc 2017-03-15 20:02:17 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0512


Note You need to log in before you can comment on or make changes to this bug.