Bug 1302304

- The version of Openshift is: 

$oadm version
oadm v3.1.1.6
kubernetes v1.1.0-origin-1107-g4c8e6f4


- There are no images tagged with "v3.1.1.6" on "registry.access.redhat.com"

- The default setting in "/etc/origin/master/master-config.yaml" is:

  imageConfig:
    format: openshift3/ose-${component}:${version}
    latest: false

I'm sitting here at the customer and was not able to force to use the penshift3/ose-deployer:3.1.0.4 version!

This is also in the case 01573740 documented!

[root@paas-master1 ~]# oadm version
oadm v3.1.1.6
kubernetes v1.1.0-origin-1107-g4c8e6f4

Dear Scott.

sorry to say this but this does not work.

ansible nodes -a "sed -i 's/\${version}/v3.0.1.4/' /etc/origin/node/node-config.yaml"
ansible nodes -a "systemctl restart atomic-openshift-node"
ansible masters -a "sed -i 's/\${version}/v3.0.1.4/' /etc/origin/master/master-config.yaml"
ansible masters -a "systemctl restart atomic-openshift-master-controllers atomic-openshift-master-api"

oc deploy docker-registry --cancel
oc deploy docker-registry --latest
oc describe pod docker-registry-4-deploy

#######
Events:
  FirstSeen     LastSeen        Count   From                                    SubobjectPath                           Reason          Message
  ─────────     ────────        ─────   ────                                    ─────────────                           ──────          ───────
  11s           11s             1       {kubelet paas-node4.sicpa-ch7.dev}      implicitly required container POD       Pulling         pulling image "openshift3/ose-pod:v3.0.1.4"
  11s           11s             1       {scheduler }                                                                    Scheduled       Successfully assigned docker-registry-4-deploy to paas-node4.sicpa-ch7.dev
  7s            7s              1       {kubelet paas-node4.sicpa-ch7.dev}      implicitly required container POD       Failed          Failed to pull image "openshift3/ose-pod:v3.0.1.4": Error: image openshift3/ose-pod:v3.0.1.4 not found
  7s            7s              1       {kubelet paas-node4.sicpa-ch7.dev}                                              FailedSync      Error syncing pod, skipping: ErrImagePull
  7s            7s              1       {kubelet paas-node4.sicpa-ch7.dev}      implicitly required container POD       Back-off        Back-off pulling image "openshift3/ose-pod:v3.0.1.4"
  7s            7s              1       {kubelet paas-node4.sicpa-ch7.dev}                                              FailedSync      Error syncing pod, skipping: ImagePullBackOff
#######

[root@paas-master1 ]# oc export dc docker-registry |egrep image:
        image: registry.access.redhat.com/openshift3/ose-docker-registry:v3.1.0.4

[root@paas-master1 ]# ssh paas-node2.sicpa-ch7.dev
Last login: Wed Jan 27 17:21:13 2016 from 10.2.64.166

[root@paas-node2 ~]# docker pull openshift3/ose-pod:v3.0.1.4

Trying to pull repository registry.access.redhat.com/openshift3/ose-pod ... not found
Trying to pull repository docker.io/openshift3/ose-pod ... not found
Error: image openshift3/ose-pod:v3.0.1.4 not found

Aleks,

Your sed command switched to 'v3.0.1.4' rather than 'v3.1.0.4'. Can you please fix that and try again?

Hi scott.

thanks for the hint.

Now i was able to pull openshift3/ose-pod:v3.1.0.4 and openshift3/ose-deployer:v3.1.0.4

br aleks

okay one more

oadm router poc-router --replicas=2   --credentials='/etc/origin/master/openshift-router.kubeconfig'   --service-account=router

####
Events:
  FirstSeen     LastSeen        Count   From                                    SubobjectPath                           Reason          Message
  ─────────     ────────        ─────   ────                                    ─────────────                           ──────          ───────
  1m            1m              1       {kubelet paas-node1.sicpa-ch7.dev}      implicitly required container POD       Pulled          Container image "openshift3/ose-pod:v3.1.0.4" already present on machine
  1m            1m              1       {scheduler }                                                                    Scheduled       Successfully assigned poc-router-1-ppxnl to paas-node1.sicpa-ch7.dev
  1m            1m              1       {kubelet paas-node1.sicpa-ch7.dev}      implicitly required container POD       Created         Created with docker id f4567f56c1f8
  1m            1m              1       {kubelet paas-node1.sicpa-ch7.dev}      implicitly required container POD       Started         Started with docker id f4567f56c1f8
  1m            20s             7       {kubelet paas-node1.sicpa-ch7.dev}      spec.containers{router}                 Back-off        Back-off pulling image "openshift3/ose-haproxy-router:v3.1.1.6"
  1m            10s             4       {kubelet paas-node1.sicpa-ch7.dev}      spec.containers{router}                 Pulling         pulling image "openshift3/ose-haproxy-router:v3.1.1.6"
  1m            6s              4       {kubelet paas-node1.sicpa-ch7.dev}      spec.containers{router}                 Failed          Failed to pull image "openshift3/ose-haproxy-router:v3.1.1.6": Error: image openshift3/ose-haproxy-router:v3.1.1.6 not found
####

Aleks,

Please add the flag --images='openshift3/ose-${component}:v3.1.0.4' to your `oadm router` and `oadm registry` commands to force a specific image version. Make sure to keep the single quotes or bash will attempt to expand the variable.

You can also edit the existing deployment configs if that's easier for you than deleting and re-creating them.

See https://docs.openshift.com/enterprise/3.1/install_config/upgrades.html#upgrading-the-router
https://docs.openshift.com/enterprise/3.1/install_config/upgrades.html#upgrading-the-registry

Made comment #4 private as it had a typo, here's the amended version.

Sorry for the troubles, we're working to get the v3.1.1.6 images pushed out  ASAP. While we're working on that there's two workarounds based on whether you're installing a new environment/host or if you've already installed a v3.1.1.6 environment.
 
If you're deploying a new environment and you're using the advanced installer you can add  openshift_pkg_version='-3.1.0.4-1.git.15.5e061c3.el7aos' to the [OSEv3:vars] section of your hosts file in order to install v3.1.0.4. Please ensure  that your `openshift-ansible` git checkout is up to date. This option is not currently available via the quick installer.

[OSEv3:vars]
openshift_pkg_version='-3.1.0.4-1.git.15.5e061c3.el7aos'


 
If you've already installed alter your imageConfig.format string to substitute 'v3.1.0.4' for the string '${version}' and restart services

# sed -i 's|${version}|v3.1.0.4|' /etc/origin/master/master-config.yaml /etc/origin/node/node-config.yaml
# For non-HA deployments
# systemctl restart atomic-openshift-master atomic-openshift-node
# For HA deployments
# systemctl restart atomic-openshift-master-api atomic-openshift-master-controllers atomic-openshift-node

Additionally, if you had created a router or registry while you had 3.1.1.6 installed the version is included in those Deployment Configs and needs to be downgraded to 'v3.1.0.4' see these two links for directions on how to change those versions 

https://docs.openshift.com/enterprise/3.1/install_config/upgrades.html#upgrading-the-router
https://docs.openshift.com/enterprise/3.1/install_config/upgrades.html#upgrading-the-registry

If you have not yet created a router or registry please make sure to add --images='openshift3/ose-${component}:v3.1.0.4' to the oadm router and oadm registry commands in order to ensure you use v3.1.0.4 images. Make sure to use the single quotes to avoid bash variable expansion.


After the v3.1.0.6 images have landed please revert these changes.

# sed -i 's|v3.1.0.4|${version}|' /etc/origin/master/master-config.yaml /etc/origin/node/node-config.yaml
# For non-HA deployments
# systemctl restart atomic-openshift-master atomic-openshift-node
# For HA deployments
# systemctl restart atomic-openshift-master-api atomic-openshift-master-controllers atomic-openshift-node

All OSE v3.1.1 images are now published. If you run into any missing images please update this bug. If you find other issues with 3.1.1 un-related to image publishing please file a new bug.

If I can have a wish for the future, from customer point of view ;-)

I would suggest to pubish the images BEFORE the 'yum update ...' updates the system.

Thanks

Succeed to pull the following images, so move it verified.
registry.access.redhat.com/openshift3/ose-deployer:v3.1.1.6
registry.access.redhat.com/openshift3/ose-docker-registry:v3.1.1.6
registry.access.redhat.com/openshift3/ose-pod:v3.1.1.6
registry.access.redhat.com/openshift3/ose-haproxy-router:v3.1.1.6

Marking this closed current release as the issue was resolved on January 18th and has been subsequently validated by QE. Sorry for the troubles, we always intend to ensure images are available prior to RPM content but failed to do so for this release.