Description of problem: This project (https://github.com/RHsyseng/jenkins-on-openshift) is to show the process of updating artifacts including OpenShift templates using Jenkins. If the OpenShift template has been modified and updated in OpenShift with a `oc apply` the DeploymentConfig will fail with "ReplicationController is invalid: spec.template.spec.containers[0].image: Required value" I tested with other templates including within the openshift namespace and if `oc apply` is used that error will also occur BUT seems to resolve itself automatically. Which is not the case when using Jenkins and the project's template. This was not an issue in OCP 3.6. Found while testing with 3.7. Version-Release number of selected component (if applicable): [root@dev-cluster ~]# oc version oc v3.7.7 kubernetes v1.7.6+a08f5eeb62 features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://dev-cluster.virtomation.com:8443 openshift v3.7.7 kubernetes v1.7.6+a08f5eeb62 [root@dev-cluster ~]# rpm -qa | grep openshift atomic-openshift-excluder-3.7.7-1.git.0.e05180c.el7.noarch atomic-openshift-node-3.7.7-1.git.0.e05180c.el7.x86_64 atomic-openshift-docker-excluder-3.7.7-1.git.0.e05180c.el7.noarch tuned-profiles-atomic-openshift-node-3.7.7-1.git.0.e05180c.el7.x86_64 atomic-openshift-3.7.7-1.git.0.e05180c.el7.x86_64 atomic-openshift-sdn-ovs-3.7.7-1.git.0.e05180c.el7.x86_64 atomic-openshift-clients-3.7.7-1.git.0.e05180c.el7.x86_64 atomic-openshift-master-3.7.7-1.git.0.e05180c.el7.x86_64 How reproducible: Steps to Reproduce: 1. curl https://raw.githubusercontent.com/openshift/nodejs-ex/master/openshift/templates/nodejs.json | oc apply -f - 2. oc process nodejs-example | oc apply -f - 3. (wait until pods are available) 4. oc process -p MEMORY_LIMIT=1Gi nodejs-example | oc apply -f - Actual results: 2017-11-21 10:00:40 -0500 EST 2017-11-21 10:00:18 -0500 EST 13 nodejs-example DeploymentConfig Warning DeploymentCreationFailed deploymentconfig-controller Couldn't deploy version 2: ReplicationController "nodejs-example-2" is invalid: spec.template.spec.containers[0].image: Required value 2017-11-21 10:01:00 -0500 EST 2017-11-21 10:00:18 -0500 EST 14 nodejs-example DeploymentConfig Warning DeploymentCreationFailed deploymentconfig-controller Couldn't deploy version 2: ReplicationController "nodejs-example-2" is invalid: spec.template.spec.containers[0].image: Required value 2017-11-21 10:01:41 -0500 EST 2017-11-21 10:00:18 -0500 EST 15 nodejs-example DeploymentConfig Warning DeploymentCreationFailed deploymentconfig-controller Couldn't deploy version 2: ReplicationController "nodejs-example-2" is invalid: spec.template.spec.containers[0].image: Required value 2017-11-21 10:02:31 -0500 EST 2017-11-21 10:02:31 -0500 EST 1 nodejs-example DeploymentConfig Normal DeploymentCreated deploymentconfig-controller Created new replication controller "nodejs-example-2" for version 2 2017-11-21 10:02:31 -0500 EST 2017-11-21 10:02:31 -0500 EST 1 nodejs-example-2-deploy Pod Normal Scheduled default-scheduler Successfully assigned nodejs-example-2-deploy to dev-cluster.virtomation.com 2017-11-21 10:02:31 -0500 EST 2017-11-21 10:02:31 -0500 EST 1 nodejs-example-2-build Pod Normal Scheduled default-scheduler Successfully assigned nodejs-example-2-build to dev-cluster.virtomation.com 2017-11-21 10:02:31 -0500 EST 2017-11-21 10:02:31 -0500 EST 1 nodejs-example-2-deploy Pod Normal SuccessfulMountVolume kubelet, dev-cluster.virtomation.com MountVolume.SetUp succeeded for volume "deployer-token-5nsbt" 2017-11-21 10:02:32 -0500 EST 2017-11-21 10:02:32 -0500 EST 1 nodejs-example-2-build Pod Normal SuccessfulMountVolume kubelet, dev-cluster.virtomation.com MountVolume.SetUp succeeded for volume "crio-socket" 2017-11-21 10:02:32 -0500 EST 2017-11-21 10:02:32 -0500 EST 1 nodejs-example-2-build Pod Normal SuccessfulMountVolume kubelet, dev-cluster.virtomation.com MountVolume.SetUp succeeded for volume "buildworkdir" 2017-11-21 10:02:32 -0500 EST 2017-11-21 10:02:32 -0500 EST 1 nodejs-example-2-build Pod Normal SuccessfulMountVolume kubelet, dev-cluster.virtomation.com MountVolume.SetUp succeeded for volume "docker-socket" 2017-11-21 10:02:32 -0500 EST 2017-11-21 10:02:32 -0500 EST 1 nodejs-example-2-build Pod Normal SuccessfulMountVolume kubelet, dev-cluster.virtomation.com MountVolume.SetUp succeeded for volume "builder-dockercfg-nzzkq-push" 2017-11-21 10:02:32 -0500 EST 2017-11-21 10:02:32 -0500 EST 1 nodejs-example-2-build Pod Normal SuccessfulMountVolume kubelet, dev-cluster.virtomation.com MountVolume.SetUp succeeded for volume "builder-token-jwvcf" 2017-11-21 10:02:34 -0500 EST 2017-11-21 10:02:34 -0500 EST 1 nodejs-example-2-deploy Pod spec.containers{deployment} Normal Pulled kubelet, dev-cluster.virtomation.com Container image "registry.ops.openshift.com/openshift3/ose-deployer:v3.7.0" already present on machine 2017-11-21 10:02:34 -0500 EST 2017-11-21 10:02:34 -0500 EST 1 nodejs-example-2-deploy Pod spec.containers{deployment} Normal Created kubelet, dev-cluster.virtomation.com Created container Expected results: No replicationcontroller errors Additional info:
QE can reproduce this in v3.7.9: openshift v3.7.9 kubernetes v1.7.6+a08f5eeb62 etcd 3.2.8 After step 4 : oc describe dc nodejs-example: oc describe dc nodejs-example Name: nodejs-example Namespace: haowang1 Created: 3 minutes ago Labels: app=nodejs-example template=nodejs-example Description: Defines how to deploy the application server Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"DeploymentConfig","metadata":{"annotations":{"description":"Defines how to deploy the application server","template.alpha.openshift.io/wait-for-ready":"true"},"labels":{"app":"nodejs-example","template":"nodejs-example"},"name":"nodejs-example","namespace":"haowang1"},"spec":{"replicas":1,"selector":{"name":"nodejs-example"},"strategy":{"type":"Rolling"},"template":{"metadata":{"labels":{"name":"nodejs-example"},"name":"nodejs-example"},"spec":{"containers":[{"env":[],"image":" ","livenessProbe":{"httpGet":{"path":"/","port":8080},"initialDelaySeconds":30,"timeoutSeconds":3},"name":"nodejs-example","ports":[{"containerPort":8080}],"readinessProbe":{"httpGet":{"path":"/","port":8080},"initialDelaySeconds":3,"timeoutSeconds":3},"resources":{"limits":{"memory":"1Gi"}}}]}},"triggers":[{"imageChangeParams":{"automatic":true,"containerNames":["nodejs-example"],"from":{"kind":"ImageStreamTag","name":"nodejs-example:latest"}},"type":"ImageChange"},{"type":"ConfigChange"}]}} template.alpha.openshift.io/wait-for-ready=true Latest Version: 2 Selector: name=nodejs-example Replicas: 1 Triggers: Image(nodejs-example@latest, auto=true), Config Strategy: Rolling Template: Pod Template: Labels: name=nodejs-example Containers: nodejs-example: Image: Port: 8080/TCP Limits: memory: 1Gi Liveness: http-get http://:8080/ delay=30s timeout=3s period=10s #success=1 #failure=3 Readiness: http-get http://:8080/ delay=3s timeout=3s period=10s #success=1 #failure=3 Environment: <none> Mounts: <none> Volumes: <none> Latest Deployment: <none> Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 3m 3m 1 deploymentconfig-controller Normal DeploymentCreated Created new replication controller "nodejs-example-1" for version 1 2m 7s 16 deploymentconfig-controller Warning DeploymentCreationFailed Couldn't deploy version 2: ReplicationController "nodejs-example-2" is invalid: spec.template.spec.containers[0].image: Required value DC: { "apiVersion": "v1", "kind": "DeploymentConfig", "metadata": { "annotations": { "description": "Defines how to deploy the application server", "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"DeploymentConfig\",\"metadata\":{\"annotations\":{\"description\":\"Defines how to deploy the application server\",\"template.alpha.openshift.io/wait-for-ready\":\"true\"},\"labels\":{\"app\":\"nodejs-example\",\"template\":\"nodejs-example\"},\"name\":\"nodejs-example\",\"namespace\":\"haowang1\"},\"spec\":{\"replicas\":1,\"selector\":{\"name\":\"nodejs-example\"},\"strategy\":{\"type\":\"Rolling\"},\"template\":{\"metadata\":{\"labels\":{\"name\":\"nodejs-example\"},\"name\":\"nodejs-example\"},\"spec\":{\"containers\":[{\"env\":[],\"image\":\" \",\"livenessProbe\":{\"httpGet\":{\"path\":\"/\",\"port\":8080},\"initialDelaySeconds\":30,\"timeoutSeconds\":3},\"name\":\"nodejs-example\",\"ports\":[{\"containerPort\":8080}],\"readinessProbe\":{\"httpGet\":{\"path\":\"/\",\"port\":8080},\"initialDelaySeconds\":3,\"timeoutSeconds\":3},\"resources\":{\"limits\":{\"memory\":\"1Gi\"}}}]}},\"triggers\":[{\"imageChangeParams\":{\"automatic\":true,\"containerNames\":[\"nodejs-example\"],\"from\":{\"kind\":\"ImageStreamTag\",\"name\":\"nodejs-example:latest\"}},\"type\":\"ImageChange\"},{\"type\":\"ConfigChange\"}]}}\n", "template.alpha.openshift.io/wait-for-ready": "true" }, "creationTimestamp": "2017-11-22T08:53:09Z", "generation": 3, "labels": { "app": "nodejs-example", "template": "nodejs-example" }, "name": "nodejs-example", "namespace": "haowang1", "resourceVersion": "228082", "selfLink": "/oapi/v1/namespaces/haowang1/deploymentconfigs/nodejs-example", "uid": "90758396-cf62-11e7-b716-fa163efcf1e2" }, "spec": { "replicas": 1, "selector": { "name": "nodejs-example" }, "strategy": { "activeDeadlineSeconds": 21600, "resources": {}, "rollingParams": { "intervalSeconds": 1, "maxSurge": "25%", "maxUnavailable": "25%", "timeoutSeconds": 600, "updatePeriodSeconds": 1 }, "type": "Rolling" }, "template": { "metadata": { "creationTimestamp": null, "labels": { "name": "nodejs-example" }, "name": "nodejs-example" }, "spec": { "containers": [ { "image": " ", "imagePullPolicy": "IfNotPresent", "livenessProbe": { "failureThreshold": 3, "httpGet": { "path": "/", "port": 8080, "scheme": "HTTP" }, "initialDelaySeconds": 30, "periodSeconds": 10, "successThreshold": 1, "timeoutSeconds": 3 }, "name": "nodejs-example", "ports": [ { "containerPort": 8080, "protocol": "TCP" } ], "readinessProbe": { "failureThreshold": 3, "httpGet": { "path": "/", "port": 8080, "scheme": "HTTP" }, "initialDelaySeconds": 3, "periodSeconds": 10, "successThreshold": 1, "timeoutSeconds": 3 }, "resources": { "limits": { "memory": "1Gi" } }, "terminationMessagePath": "/dev/termination-log", "terminationMessagePolicy": "File" } ], "dnsPolicy": "ClusterFirst", "restartPolicy": "Always", "schedulerName": "default-scheduler", "securityContext": {}, "terminationGracePeriodSeconds": 30 } }, "test": false, "triggers": [ { "imageChangeParams": { "automatic": true, "containerNames": [ "nodejs-example" ], "from": { "kind": "ImageStreamTag", "name": "nodejs-example:latest", "namespace": "haowang1" } }, "type": "ImageChange" }, { "type": "ConfigChange" } ] }, "status": { "availableReplicas": 1, "conditions": [ { "lastTransitionTime": "2017-11-22T08:54:02Z", "lastUpdateTime": "2017-11-22T08:54:02Z", "message": "Deployment config has minimum availability.", "status": "True", "type": "Available" }, { "lastTransitionTime": "2017-11-22T08:54:17Z", "lastUpdateTime": "2017-11-22T08:54:17Z", "message": "ReplicationController \"nodejs-example-2\" is invalid: spec.template.spec.containers[0].image: Required value", "reason": "ReplicationControllerCreateError", "status": "False", "type": "Progressing" } ], "details": { "causes": [ { "imageTrigger": { "from": { "kind": "DockerImage", "name": " " } }, "type": "ImageChange" } ], "message": "image change" }, "latestVersion": 2, "observedGeneration": 3, "readyReplicas": 1, "replicas": 1, "unavailableReplicas": 0, "updatedReplicas": 0 } } ], "kind": "List", "metadata": { "resourceVersion": "", "selfLink": "" }
PR for master branch: https://github.com/openshift/origin/pull/17539
PR was merged, moving to QA.
Verified with: openshift v3.9.0-0.23.0 kubernetes v1.9.1+a0ce1bc657 etcd 3.2.8
Failed with: openshift v3.7.23 kubernetes v1.7.6+a08f5eeb62 etcd 3.2.8 I still see the event : #oc describe dc FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 50m 50m 1 deploymentconfig-controller Normal DeploymentCreated Created new replication controller "nodejs-example-1" for version 1 41m 38m 16 deploymentconfig-controller Warning DeploymentCreationFailed
if QA failed for 3.7, 3.7 BZ is here https://bugzilla.redhat.com/show_bug.cgi?id=1546087 This should likely stay verified for 3.9
Sorry, comments paste to the wrong bug, this should failed 3.7.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:3748