Bug 1482824 - Upgrade playbook to 3.6 failed by error with Pod "XXX" is invalid
Summary: Upgrade playbook to 3.6 failed by error with Pod "XXX" is invalid
Keywords:
Status: CLOSED DUPLICATE of bug 1383707
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Node
Version: 3.6.1
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: Michal Fojtik
QA Contact: DeShuai Ma
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-08-18 07:45 UTC by Takayoshi Tanaka
Modified: 2017-08-25 16:25 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-08-25 16:25:39 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Takayoshi Tanaka 2017-08-18 07:45:42 UTC
Description of problem:
During executing upgrade playbook failed.

Version-Release number of selected component (if applicable):
3.6

How reproducible:
always in my environment.

Steps to Reproduce:
1. I have my test environment OCP 3.5.
2. Preparing for an Automated Upgrade
https://docs.openshift.com/container-platform/3.6/install_config/upgrading/automated_upgrades.html#preparing-for-an-automated-upgrade
3. Execute playbook.
# ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_6/upgrade.yml 
https://docs.openshift.com/container-platform/3.6/install_config/upgrading/automated_upgrades.html#running-the-upgrade-playbook-directly

Actual results:
Upgrade without error.

Expected results:
Failed with errors.
```
TASK [Upgrade all storage] ****************************************************************************************************************************************************************************************
fatal: [tatanaka-ose3-single.usersys.redhat.com]: FAILED! => {
    "changed": true, 
    "cmd": [
        "oc", 
        "adm", 
        "--config=/etc/origin/master/admin.kubeconfig", 
        "migrate", 
        "storage", 
        "--include=*", 
        "--confirm"
    ], 
    "delta": "0:00:45.593403", 
    "end": "2017-08-16 21:20:36.105035", 
    "failed": true, 
    "failed_when_result": true, 
    "rc": 1, 
    "start": "2017-08-16 21:19:50.511632"
}

STDOUT:

error:     pods/aspnet-fileresult-2-build -n aspnet: Pod "aspnet-fileresult-2-build" is invalid: spec: Forbidden: pod updates may not change fields other than `containers[*].image` or `spec.activeDeadlineSeconds`
error:     pods/dotnet-debug-3-gvnwz -n aspnet: Pod "dotnet-debug-3-gvnwz" is invalid: spec: Forbidden: pod updates may not change fields other than `containers[*].image` or `spec.activeDeadlineSeconds`

//also same errors for other pods
summary: total=2417 errors=278 ignored=0 unchanged=2136 migrated=3
info: to rerun only failing resources, add --include=pods
error: 278 resources failed to migrate


STDERR:

error: exit directly

```


Additional info:
Some pods have been migrated without error and others got the above error. I have no idea which pods got the error.  

The errored pod has below pod definition.

```
apiVersion: v1
kind: Pod
metadata:
  name: aspnet-fileresult-2-build
  namespace: aspnet
  selfLink: /api/v1/namespaces/aspnet/pods/aspnet-fileresult-2-build
  uid: b03952cd-1daf-11e7-bf9d-001a4a40dc6f
  resourceVersion: '4521536'
  creationTimestamp: '2017-04-10T05:36:47Z'
  labels:
    openshift.io/build.name: aspnet-fileresult-2
  annotations:
    openshift.io/build.name: aspnet-fileresult-2
    openshift.io/scc: privileged
spec:
  volumes:
    - name: docker-socket
      hostPath:
        path: /var/run/docker.sock
    - name: builder-dockercfg-5auz1-push
      secret:
        secretName: builder-dockercfg-5auz1
        defaultMode: 420
    - name: builder-token-kgywl
      secret:
        secretName: builder-token-kgywl
        defaultMode: 420
  containers:
    - name: sti-build
      image: 'openshift3/ose-sti-builder:v3.4.0.40'
      args:
        - '--loglevel=0'
      env:
        - name: BUILD
          value: >
            {"kind":"Build","apiVersion":"v1","metadata":{"name":"aspnet-fileresult-2","namespace":"aspnet","selfLink":"/oapi/v1/namespaces/aspnet/builds/aspnet-fileresult-2","uid":"b03325e4-1daf-11e7-bf9d-001a4a40dc6f","resourceVersion":"4521129","creationTimestamp":"2017-04-10T05:36:47Z","labels":{"app":"aspnet-fileresult","buildconfig":"aspnet-fileresult","openshift.io/build-config.name":"aspnet-fileresult","openshift.io/build.start-policy":"Serial","template":"aspnet-s2i"},"annotations":{"openshift.io/build-config.name":"aspnet-fileresult","openshift.io/build.number":"2"}},"spec":{"serviceAccount":"builder","source":{"type":"Git","git":{"uri":"https://github.com/tanaka-takayoshi/s2i-aspnet-example.git","ref":"StaticFile","httpProxy":"http://my.proxy:8080","httpsProxy":"https://my.proxy:8443","noProxy":"github.com"},"contextDir":"app"},"strategy":{"type":"Source","sourceStrategy":{"from":{"kind":"DockerImage","name":"registry.access.redhat.com/dotnet/dotnetcore-10-rhel7@sha256:7b58ba07ac2c32f76e4ac112292837ec5091544c9011214fd8c46130d86966eb"}}},"output":{"to":{"kind":"DockerImage","name":"172.30.184.255:5000/aspnet/aspnet-fileresult:latest"},"pushSecret":{"name":"builder-dockercfg-5auz1"}},"resources":{},"postCommit":{},"nodeSelector":null,"triggeredBy":[{"message":"Image
            change","imageChangeBuild":{"imageID":"registry.access.redhat.com/dotnet/dotnetcore-10-rhel7@sha256:7b58ba07ac2c32f76e4ac112292837ec5091544c9011214fd8c46130d86966eb","fromRef":{"kind":"ImageStreamTag","name":"s2i-aspnet:latest"}}}]},"status":{"phase":"New","outputDockerImageReference":"172.30.184.255:5000/aspnet/aspnet-fileresult:latest","config":{"kind":"BuildConfig","namespace":"aspnet","name":"aspnet-fileresult"}}}
        - name: SOURCE_REPOSITORY
          value: 'https://github.com/tanaka-takayoshi/s2i-aspnet-example.git'
        - name: SOURCE_URI
          value: 'https://github.com/tanaka-takayoshi/s2i-aspnet-example.git'
        - name: SOURCE_CONTEXT_DIR
          value: app
        - name: SOURCE_REF
          value: StaticFile
        - name: ORIGIN_VERSION
          value: v3.4.0.40
        - name: ALLOWED_UIDS
          value: 1-
        - name: DROP_CAPS
          value: 'KILL,MKNOD,SETGID,SETUID,SYS_CHROOT'
        - name: PUSH_DOCKERCFG_PATH
          value: /var/run/secrets/openshift.io/push
      resources: {}
      volumeMounts:
        - name: docker-socket
          mountPath: /var/run/docker.sock
        - name: builder-dockercfg-5auz1-push
          readOnly: true
          mountPath: /var/run/secrets/openshift.io/push
        - name: builder-token-kgywl
          readOnly: true
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      terminationMessagePath: /dev/termination-log
      imagePullPolicy: IfNotPresent
      securityContext:
        privileged: true
  restartPolicy: Never
  terminationGracePeriodSeconds: 30
  dnsPolicy: ClusterFirst
  serviceAccountName: builder
  serviceAccount: builder
  nodeName: tatanaka-ose3-single.usersys.redhat.com
  securityContext: {}
  imagePullSecrets:
    - name: builder-dockercfg-5auz1
status:
  phase: Succeeded
  conditions:
    - type: Initialized
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2017-04-10T05:36:47Z'
      reason: PodCompleted
    - type: Ready
      status: 'False'
      lastProbeTime: null
      lastTransitionTime: '2017-04-10T05:42:11Z'
      reason: PodCompleted
    - type: PodScheduled
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2017-04-10T05:36:47Z'
  hostIP: 10.64.221.138
  podIP: 10.1.0.76
  startTime: '2017-04-10T05:36:47Z'
  containerStatuses:
    - name: sti-build
      state:
        terminated:
          exitCode: 0
          reason: Completed
          startedAt: '2017-04-10T05:37:01Z'
          finishedAt: '2017-04-10T05:42:06Z'
          containerID: >-
            docker://f6a6829f7cf70834bf6f1566e357322640dccac66731fe28e97cac1a4c2f5d45
      lastState: {}
      ready: false
      restartCount: 0
      image: 'openshift3/ose-sti-builder:v3.4.0.40'
      imageID: >-
        docker-pullable://registry.access.redhat.com/openshift3/ose-sti-builder@sha256:5637c7feaec481ecd68f337aa7982d510df1faac18608565883cd6670b0756f2
      containerID: >-
        docker://f6a6829f7cf70834bf6f1566e357322640dccac66731fe28e97cac1a4c2f5d45
```

Comment 1 Jordan Liggitt 2017-08-22 15:12:08 UTC
seems likely to be a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1383707

Comment 2 Jordan Liggitt 2017-08-25 16:25:39 UTC

*** This bug has been marked as a duplicate of bug 1383707 ***


Note You need to log in before you can comment on or make changes to this bug.