Bug 1302304 - OSE 3.1.1 no docker images available in the public registry
OSE 3.1.1 no docker images available in the public registry
Status: CLOSED CURRENTRELEASE
Product: OpenShift Container Platform
Classification: Red Hat
Component: Release (Show other bugs)
3.1.0
Unspecified Unspecified
urgent Severity urgent
: ---
: ---
Assigned To: Greg Sterling
libra bugs
:
Depends On:
Blocks: 1267746
  Show dependency treegraph
 
Reported: 2016-01-27 07:48 EST by Alexander Koksharov
Modified: 2016-02-04 09:20 EST (History)
14 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-02-04 09:20:50 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
jhonce: needinfo-


Attachments (Terms of Use)

  None (edit)
Description Alexander Koksharov 2016-01-27 07:48:33 EST
Description of problem:
oc get ev | grep docker-registry-3-deploy
14s         14s        1         docker-registry-3-deploy   Pod                                                         Scheduled           {scheduler }                            Successfully assigned docker-registry-3-deploy to ose-allinone.example.com
14s         14s        1         docker-registry-3-deploy   Pod                     implicitly required container POD   Pulling             {kubelet ose-allinone.example.com}      pulling image "openshift3/ose-pod:v3.1.1.6"
8s          8s         1         docker-registry-3-deploy   Pod                     implicitly required container POD   Failed              {kubelet ose-allinone.example.com}      Failed to pull image "openshift3/ose-pod:v3.1.1.6": Error: image openshift3/ose-pod:v3.1.1.6 not found
8s          8s         1         docker-registry-3-deploy   Pod                                                         FailedSync          {kubelet ose-allinone.example.com}      Error syncing pod, skipping: ErrImagePull
6s          6s         1         docker-registry-3-deploy   Pod                     implicitly required container POD   Back-off            {kubelet ose-allinone.example.com}      Back-off pulling image "openshift3/ose-pod:v3.1.1.6"
6s          6s         1         docker-registry-3-deploy   Pod                                                         FailedSync          {kubelet ose-allinone.example.com}      Error syncing pod, skipping: ImagePullBackOff

Version-Release number of selected component (if applicable):
OSE 3.1.1

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 2 Ryan Howe 2016-01-27 09:50:17 EST
- The version of Openshift is: 

$oadm version
oadm v3.1.1.6
kubernetes v1.1.0-origin-1107-g4c8e6f4


- There are no images tagged with "v3.1.1.6" on "registry.access.redhat.com"

- The default setting in "/etc/origin/master/master-config.yaml" is:

  imageConfig:
    format: openshift3/ose-${component}:${version}
    latest: false
Comment 3 Aleks Lazic 2016-01-27 10:48:54 EST
I'm sitting here at the customer and was not able to force to use the penshift3/ose-deployer:3.1.0.4 version!

This is also in the case 01573740 documented!

[root@paas-master1 ~]# oadm version
oadm v3.1.1.6
kubernetes v1.1.0-origin-1107-g4c8e6f4
Comment 5 Aleks Lazic 2016-01-27 11:45:15 EST
Dear Scott.

sorry to say this but this does not work.

ansible nodes -a "sed -i 's/\${version}/v3.0.1.4/' /etc/origin/node/node-config.yaml"
ansible nodes -a "systemctl restart atomic-openshift-node"
ansible masters -a "sed -i 's/\${version}/v3.0.1.4/' /etc/origin/master/master-config.yaml"
ansible masters -a "systemctl restart atomic-openshift-master-controllers atomic-openshift-master-api"

oc deploy docker-registry --cancel
oc deploy docker-registry --latest
oc describe pod docker-registry-4-deploy

#######
Events:
  FirstSeen     LastSeen        Count   From                                    SubobjectPath                           Reason          Message
  ─────────     ────────        ─────   ────                                    ─────────────                           ──────          ───────
  11s           11s             1       {kubelet paas-node4.sicpa-ch7.dev}      implicitly required container POD       Pulling         pulling image "openshift3/ose-pod:v3.0.1.4"
  11s           11s             1       {scheduler }                                                                    Scheduled       Successfully assigned docker-registry-4-deploy to paas-node4.sicpa-ch7.dev
  7s            7s              1       {kubelet paas-node4.sicpa-ch7.dev}      implicitly required container POD       Failed          Failed to pull image "openshift3/ose-pod:v3.0.1.4": Error: image openshift3/ose-pod:v3.0.1.4 not found
  7s            7s              1       {kubelet paas-node4.sicpa-ch7.dev}                                              FailedSync      Error syncing pod, skipping: ErrImagePull
  7s            7s              1       {kubelet paas-node4.sicpa-ch7.dev}      implicitly required container POD       Back-off        Back-off pulling image "openshift3/ose-pod:v3.0.1.4"
  7s            7s              1       {kubelet paas-node4.sicpa-ch7.dev}                                              FailedSync      Error syncing pod, skipping: ImagePullBackOff
#######

[root@paas-master1 ]# oc export dc docker-registry |egrep image:
        image: registry.access.redhat.com/openshift3/ose-docker-registry:v3.1.0.4

[root@paas-master1 ]# ssh paas-node2.sicpa-ch7.dev
Last login: Wed Jan 27 17:21:13 2016 from 10.2.64.166

[root@paas-node2 ~]# docker pull openshift3/ose-pod:v3.0.1.4

Trying to pull repository registry.access.redhat.com/openshift3/ose-pod ... not found
Trying to pull repository docker.io/openshift3/ose-pod ... not found
Error: image openshift3/ose-pod:v3.0.1.4 not found
Comment 6 Scott Dodson 2016-01-27 11:52:05 EST
Aleks,

Your sed command switched to 'v3.0.1.4' rather than 'v3.1.0.4'. Can you please fix that and try again?
Comment 7 Aleks Lazic 2016-01-27 12:08:19 EST
Hi scott.

thanks for the hint.

Now i was able to pull openshift3/ose-pod:v3.1.0.4 and openshift3/ose-deployer:v3.1.0.4

br aleks
Comment 8 Aleks Lazic 2016-01-27 12:30:02 EST
okay one more

oadm router poc-router --replicas=2   --credentials='/etc/origin/master/openshift-router.kubeconfig'   --service-account=router

####
Events:
  FirstSeen     LastSeen        Count   From                                    SubobjectPath                           Reason          Message
  ─────────     ────────        ─────   ────                                    ─────────────                           ──────          ───────
  1m            1m              1       {kubelet paas-node1.sicpa-ch7.dev}      implicitly required container POD       Pulled          Container image "openshift3/ose-pod:v3.1.0.4" already present on machine
  1m            1m              1       {scheduler }                                                                    Scheduled       Successfully assigned poc-router-1-ppxnl to paas-node1.sicpa-ch7.dev
  1m            1m              1       {kubelet paas-node1.sicpa-ch7.dev}      implicitly required container POD       Created         Created with docker id f4567f56c1f8
  1m            1m              1       {kubelet paas-node1.sicpa-ch7.dev}      implicitly required container POD       Started         Started with docker id f4567f56c1f8
  1m            20s             7       {kubelet paas-node1.sicpa-ch7.dev}      spec.containers{router}                 Back-off        Back-off pulling image "openshift3/ose-haproxy-router:v3.1.1.6"
  1m            10s             4       {kubelet paas-node1.sicpa-ch7.dev}      spec.containers{router}                 Pulling         pulling image "openshift3/ose-haproxy-router:v3.1.1.6"
  1m            6s              4       {kubelet paas-node1.sicpa-ch7.dev}      spec.containers{router}                 Failed          Failed to pull image "openshift3/ose-haproxy-router:v3.1.1.6": Error: image openshift3/ose-haproxy-router:v3.1.1.6 not found
####
Comment 9 Scott Dodson 2016-01-27 12:57:32 EST
Aleks,

Please add the flag --images='openshift3/ose-${component}:v3.1.0.4' to your `oadm router` and `oadm registry` commands to force a specific image version. Make sure to keep the single quotes or bash will attempt to expand the variable.

You can also edit the existing deployment configs if that's easier for you than deleting and re-creating them.

See https://docs.openshift.com/enterprise/3.1/install_config/upgrades.html#upgrading-the-router
https://docs.openshift.com/enterprise/3.1/install_config/upgrades.html#upgrading-the-registry
Comment 10 Scott Dodson 2016-01-27 12:59:21 EST
Made comment #4 private as it had a typo, here's the amended version.

Sorry for the troubles, we're working to get the v3.1.1.6 images pushed out  ASAP. While we're working on that there's two workarounds based on whether you're installing a new environment/host or if you've already installed a v3.1.1.6 environment.
 
If you're deploying a new environment and you're using the advanced installer you can add  openshift_pkg_version='-3.1.0.4-1.git.15.5e061c3.el7aos' to the [OSEv3:vars] section of your hosts file in order to install v3.1.0.4. Please ensure  that your `openshift-ansible` git checkout is up to date. This option is not currently available via the quick installer.

[OSEv3:vars]
openshift_pkg_version='-3.1.0.4-1.git.15.5e061c3.el7aos'


 
If you've already installed alter your imageConfig.format string to substitute 'v3.1.0.4' for the string '${version}' and restart services

# sed -i 's|${version}|v3.1.0.4|' /etc/origin/master/master-config.yaml /etc/origin/node/node-config.yaml
# For non-HA deployments
# systemctl restart atomic-openshift-master atomic-openshift-node
# For HA deployments
# systemctl restart atomic-openshift-master-api atomic-openshift-master-controllers atomic-openshift-node

Additionally, if you had created a router or registry while you had 3.1.1.6 installed the version is included in those Deployment Configs and needs to be downgraded to 'v3.1.0.4' see these two links for directions on how to change those versions 

https://docs.openshift.com/enterprise/3.1/install_config/upgrades.html#upgrading-the-router
https://docs.openshift.com/enterprise/3.1/install_config/upgrades.html#upgrading-the-registry

If you have not yet created a router or registry please make sure to add --images='openshift3/ose-${component}:v3.1.0.4' to the oadm router and oadm registry commands in order to ensure you use v3.1.0.4 images. Make sure to use the single quotes to avoid bash variable expansion.


After the v3.1.0.6 images have landed please revert these changes.

# sed -i 's|v3.1.0.4|${version}|' /etc/origin/master/master-config.yaml /etc/origin/node/node-config.yaml
# For non-HA deployments
# systemctl restart atomic-openshift-master atomic-openshift-node
# For HA deployments
# systemctl restart atomic-openshift-master-api atomic-openshift-master-controllers atomic-openshift-node
Comment 15 Scott Dodson 2016-01-28 14:18:51 EST
All OSE v3.1.1 images are now published. If you run into any missing images please update this bug. If you find other issues with 3.1.1 un-related to image publishing please file a new bug.
Comment 16 Aleks Lazic 2016-01-29 03:50:55 EST
If I can have a wish for the future, from customer point of view ;-)

I would suggest to pubish the images BEFORE the 'yum update ...' updates the system.

Thanks
Comment 17 Johnny Liu 2016-02-04 03:26:14 EST
Succeed to pull the following images, so move it verified.
registry.access.redhat.com/openshift3/ose-deployer:v3.1.1.6
registry.access.redhat.com/openshift3/ose-docker-registry:v3.1.1.6
registry.access.redhat.com/openshift3/ose-pod:v3.1.1.6
registry.access.redhat.com/openshift3/ose-haproxy-router:v3.1.1.6
Comment 18 Scott Dodson 2016-02-04 09:20:50 EST
Marking this closed current release as the issue was resolved on January 18th and has been subsequently validated by QE. Sorry for the troubles, we always intend to ensure images are available prior to RPM content but failed to do so for this release.

Note You need to log in before you can comment on or make changes to this bug.