Bug 1566328 - Upgrade from 3.7 to 3.9 fail at the Task [Upgrade all storage] for servicebinding (fix for 3.10)
Summary: Upgrade from 3.7 to 3.9 fail at the Task [Upgrade all storage] for servicebin...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Service Catalog
Version: 3.9.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 3.10.0
Assignee: Jeff Peeler
QA Contact: Jian Zhang
URL:
Whiteboard:
Depends On:
Blocks: 1586135
TreeView+ depends on / blocked
 
Reported: 2018-04-12 05:13 UTC by Sudarshan Chaudhari
Modified: 2018-07-30 19:13 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
undefined
Clone Of:
: 1586135 (view as bug list)
Environment:
Last Closed: 2018-07-30 19:12:49 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:1816 None None None 2018-07-30 19:13:33 UTC

Description Sudarshan Chaudhari 2018-04-12 05:13:43 UTC
Description of problem:

The automated In-Place upgrade playbook is failing while migrating the servicebindig


Version-Release number of the following components:

# rpm -qa openshift-ansible
openshift-ansible-3.9.14-1.git.3.c62bc34.el7.noarch

Run the Automated In-Place Upgrade Playbook.
# ansible-playbook -i </path/to/inventory/file> /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_9/upgrade.yml -vvv


Failed with error:

<os-master1.example.com> (0, '\r\n{"changed": true, "end": "2018-04-06 13:04:08.719025", "stdout": "E0406 13:03:59.282720 error:     -n portal-dev servicebindings/eap64-basic-s2i-qnh9f-7tb2z: ServiceBinding.servicecatalog.k8s.io \\"eap64-basic-s2i-qnh9f-7tb2z\\" is invalid: status.unbindStatus: Unsupported value: \\"\\": supported values: \\"NotRequired\\", \\"Required\\", \\"Succeeded\\", \\"Failed\\"\\nsummary: total=4994 errors=1 ignored=0 unchanged=4992 migrated=1\\ninfo: to rerun only failing resources, add --include=servicebindings\\nerror: 1 resources failed to migrate", "cmd": ["oc", "adm", "--config=/etc/origin/master/admin.kubeconfig", "migrate", "storage", "--include=*", "--confirm"], "failed": true, "delta": "0:01:52.488851", "stderr": "", "rc": 1, "invocation": {"module_args": {"warn": true, "executable": null, "_uses_shell": false, "_raw_params": "oc adm --config=/etc/origin/master/admin.kubeconfig migrate storage --include=* --confirm", "removes": null, "creates": null, "chdir": null, "stdin": null}}, "start": "2018-04-06 13:02:16.230174", "msg": "non-zero return code"}\r\n', 'Shared connection to os-master1.example.com closed.\r\n')
fatal: [os-master1.example.com]: FAILED! => {
    "changed": true,
    "cmd": [
        "oc",
        "adm",
        "--config=/etc/origin/master/admin.kubeconfig", 
        "migrate", 
        "storage", 
        "--include=*", 
        "--confirm"
    ], 
    "delta": "0:01:52.488851", 
    "end": "2018-04-06 13:04:08.719025", 
    "failed_when_result": true, 
    "invocation": {
        "module_args": {
            "_raw_params": "oc adm --config=/etc/origin/master/admin.kubeconfig migrate storage --include=* --confirm", 
            "_uses_shell": false, 
            "chdir": null,
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "warn": true
        }
    },
    "msg": "non-zero return code",
    "rc": 1,
    "start": "2018-04-06 13:02:16.230174",
    "stderr": "",
    "stderr_lines": [],
    "stdout": "E0406 13:03:59.282720 error:     -n portal-dev servicebindings/eap64-basic-s2i-qnh9f-7tb2z: ServiceBinding.servicecatalog.k8s.io \"eap64-basic-s2i-qnh9f-7tb2z\" is invalid: status.unbindStatus: Unsupported value: \"\": supported values: \"NotRequired\", \"Required\", \"Succeeded\", \"Failed\"\nsummary: total=4994 errors=1 ignored=0 unchanged=4992 migrated=1\ninfo: to rerun only failing resources, add --include=servicebindings\nerror: 1 resources failed to migrate",
    "stdout_lines": [
        "E0406 13:03:59.282720 error:     -n portal-dev servicebindings/eap64-basic-s2i-qnh9f-7tb2z: ServiceBinding.servicecatalog.k8s.io \"eap64-basic-s2i-qnh9f-7tb2z\" is invalid: status.unbindStatus: Unsupported value: \"\": supported values: \"NotRequired\", \"Required\", \"Succeeded\", \"Failed\"",
        "summary: total=4994 errors=1 ignored=0 unchanged=4992 migrated=1",
        "info: to rerun only failing resources, add --include=servicebindings",
        "error: 1 resources failed to migrate"
    ]
}
        to retry, use: --limit @/usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_9/upgrade.retry


                                                                                                                                 
When manually ran the command: 
:>oc adm migrate storage --include=* --confirm

E0404 18:53:15.075747 error:     -n portal-dev servicebindings/eap64-basic-s2i-qnh9f-7tb2z: ServiceBinding.servicecatalog.k8s.io "eap64-basic-s2i-qnh9f-7tb2z" is invalid: st
atus.unbindStatus: Unsupported value: "": supported values: "Succeeded", "Failed", "NotRequired", "Required"
summary: total=5001 errors=1 ignored=0 unchanged=4998 migrated=2
info: to rerun only failing resources, add --include=servicebindings
error: 1 resources failed to migrate

Actual results:
Please include the entire output from the last TASK line through the end of output if an error is generated


Expected results:
Then upgrade should be successfull.

Comment 4 Paul Morie 2018-04-19 21:02:14 UTC
It looks like this is happening because there was a validation added for this field, but no default for resources where it is unset.

Comment 5 Paul Morie 2018-04-19 22:33:36 UTC
Upstream PR that introduced this bug: https://github.com/kubernetes-incubator/service-catalog/pull/1544

I think we need to:

- Relax the validation on update
- Add a conversion that sets the field value to the right value (NotRequired) if it is unset

Comment 6 Paul Morie 2018-04-19 23:04:07 UTC
Created an issue for this upstream: https://github.com/kubernetes-incubator/service-catalog/issues/1960

Comment 8 Jian Zhang 2018-04-24 06:09:39 UTC
Sudarshan,

For this scenario, you can delete the `finalizers` field of the `servicebindings` first and have a try. For example:

#oc edit servicebinding  rh-postgresql-apb-xcrd4-f6vgc (PS: delete the below two lines.)

  finalizers:
  - kubernetes-incubator/service-catalog

Comment 9 Sudarshan Chaudhari 2018-04-24 06:43:29 UTC
Thanks for the update, 

I will try this and will get back to you.

Comment 10 Jeff Peeler 2018-04-26 19:43:08 UTC
Upstream PR: https://github.com/kubernetes-incubator/service-catalog/pull/1972

Comment 26 Jeff Peeler 2018-06-05 15:52:51 UTC
The upstream PR in comment 10 was merged upstream in 0.1.17 and is already available in openshift for 3.10.

Comment 31 Jian Zhang 2018-06-07 10:01:07 UTC
@Jeff

Ok, I see, thank you for your clarification!

Comment 37 errata-xmlrpc 2018-07-30 19:12:49 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1816


Note You need to log in before you can comment on or make changes to this bug.