Bug 1554645 - OpenShift verison print not correctly cause the installation failed
Summary: OpenShift verison print not correctly cause the installation failed
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Release
Version: 3.3.1
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 3.3.1
Assignee: Justin Pierce
QA Contact: Chuan Yu
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-03-13 05:38 UTC by Chuan Yu
Modified: 2018-04-18 07:02 UTC (History)
17 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-04-18 07:00:36 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 3389911 0 None None None 2018-03-22 14:16:28 UTC
Red Hat Product Errata RHBA-2018:1134 0 None None None 2018-04-18 07:02:00 UTC

Description Chuan Yu 2018-03-13 05:38:04 UTC
Description of problem:
Openshift version print not correctly,this cause the installer failed.

Version-Release number of selected component (if applicable):
openshift v3.3.1.46.11
kubernetes v1.3.0+52492b4
etcd 2.3.0+git

How reproducible:
always

Steps to Reproduce:
1.openshift verison
2.
3.

Actual results:
openshift v3.3.1.46.11+e236015
kubernetes v1.3.0+52492b4
etcd 2.3.0+git

Expected results:
the openshift version should no suffix +e236015

Additional info:

TASK [openshift_version : Set precise containerized version to configure if openshift_release specified] ***
Tuesday 13 March 2018  03:18:32 +0000 (0:00:00.018)       0:00:32.341 ********* 

changed: [host......] => {
    "changed": true, 
    "cmd": [
        "docker", 
        "run", 
        "--rm", 
        "openshift3/ose:v3.3", 
        "version"
    ], 
    "delta": "0:00:49.844265", 
    "end": "2018-03-13 03:19:22.347698", 
    "rc": 0, 
    "start": "2018-03-13 03:18:32.503433"
}

STDOUT:

openshift v3.3.1.46.11+e236015
kubernetes v1.3.0+52492b4
etcd 2.3.0+git


TASK [openshift_cli : Pull CLI Image] ******************************************
Tuesday 13 March 2018  03:20:50 +0000 (0:00:00.026)       0:02:50.393 ********* 
fatal: [host......]: FAILED! => {
    "changed": false, 
    "cmd": [
        "docker", 
        "pull", 
        "openshift3/ose:v3.3.1.46.11+e236015"
    ], 
    "delta": "0:00:00.029532", 
    "end": "2018-03-13 03:20:50.576362", 
    "failed": true, 
    "rc": 1, 
    "start": "2018-03-13 03:20:50.546830"
}

STDERR:

Error parsing reference: "openshift3/ose:v3.3.1.46.11+e236015" is not a valid repository/tag

Comment 2 Russell Teague 2018-03-13 13:16:29 UTC
A commit[1] was added in 3.4 to remove any trailing commit hash from the version returned when obtaining the installed version from the master.  This was an edge case where a build was made from an untagged version.

[1] https://github.com/openshift/openshift-ansible/commit/54820399a202a687b0251febbab64fb590577234

Comment 12 Russell Teague 2018-03-15 20:23:39 UTC
In order to force the correct version these additional inventory vars should be defined:

openshift_release=v3.3
openshift_image_tag=v3.3.1.46.11
openshift_pkg_version=-3.3.1.46.11
openshift_protect_installed_version=False
oreg_url=openshift3/ose-${component}:v3.3.1.46.11

openshift_image_tag and openshift_pkg_version are specified so the installer does not attempt to gather specific versions by other means.

openshift_protect_installed_version is set to false to ensure the version reported by the installed openshift binary is not used.

oreg_url is specified using the required version to prevent openshift from defaulting to the installed version which is invalid given the additional commit offset.

openshift_image_tag and openshift_pkg_version are in the example inventory in 3.3 but had not yet made it into the online documentation.
https://github.com/openshift/openshift-ansible/blob/release-1.3/inventory/byo/hosts.ose.example#L35-L43

These changes should be temporary until the 3.3 packages/images are rebuilt with corrected version strings.

Comment 21 Junqi Zhao 2018-03-21 07:34:11 UTC
Tested today, installed by rpm, still wrongly treat v3.3.1.46.11+e236015 as image version

# oc get po -n default
NAME                        READY     STATUS              RESTARTS   AGE
docker-registry-8-deploy    0/1       ContainerCreating   0          17m
registry-console-3-deploy   0/1       ContainerCreating   0          17m
router-3-deploy             0/1       ContainerCreating   0          17m

#  oc describe po docker-registry-8-deploy -n default
Events:
  FirstSeen	LastSeen	Count	From								SubobjectPath	Type		Reason		Message
  ---------	--------	-----	----								-------------	--------	------		-------
  18m		18m		1	{default-scheduler }								Normal		Scheduled	Successfully assigned docker-registry-8-deploy to host-8-241-87.host.centralci.eng.rdu2.redhat.com
  18m		3m		8	{kubelet host-8-241-87.host.centralci.eng.rdu2.redhat.com}			Warning		FailedSync	Error syncing pod, skipping: failed to "StartContainer" for "POD" with ErrImagePull: "couldn't parse image reference \"brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/ose-pod:v3.3.1.46.11+e236015\": invalid reference format"

  18m	6s	77	{kubelet host-8-241-87.host.centralci.eng.rdu2.redhat.com}		Warning	FailedSync	Error syncing pod, skipping: failed to "StartContainer" for "POD" with ImagePullBackOff: "Back-off pulling image \"brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/ose-pod:v3.3.1.46.11+e236015\""

Comment 23 Russell Teague 2018-03-21 13:58:30 UTC
For new installs, the workaround for inventory variables provided in comment 12 should be used.

For an existing installation experiencing problems with deployments, to temporarily work around the invalid version provided by the RPMs and images, the following steps should be performed.

On all masters:
sudo sed -i.bak 's/\${version}/v3.3.1.46.11/g' /etc/origin/master/master-config.yaml
sudo systemctl restart atomic-openshift-master.service

On all nodes:
sudo sed -i.bak 's/\${version}/v3.3.1.46.11/g' /etc/origin/node/node-config.yaml
sudo systemctl restart atomic-openshift-node.service

NOTE: The above changes should be reverted when a new version of openshift is installed which no longer has the trailing commit offset.

After the service config files have been updated and the services restarted, any existing deployment configs should be modified to ensure the correct version is specified without any trailing commit offset.  Examples are provided below.

oc edit dc/docker-registry

        image: openshift3/ose-docker-registry:v3.3.1.46.11

oc edit dc/router

        image: openshift3/ose-haproxy-router:v3.3.1.46.11

Editing the deployment configs will cause the deployment to restart and use the new version.

For other deployment configs that do not need the image setting updated, the deployment will need to be restarted.  For example:

oc deploy --cancel dc/registry-console
oc deploy --latest dc/registry-console

Comment 28 Chuan Yu 2018-04-08 02:25:22 UTC
Verified.
# openshift version
openshift v3.3.1.46.39
kubernetes v1.3.0+52492b4
etcd 2.3.0+git

Comment 31 errata-xmlrpc 2018-04-18 07:00:36 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1134


Note You need to log in before you can comment on or make changes to this bug.