Bug 1852753
| Summary: | Upgrade from 3.10 to 3.11 failing with error: Error: unknown flag: --field-selector\n\n\n | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Oscar Casal Sanchez <ocasalsa> |
| Component: | Installer | Assignee: | Russell Teague <rteague> |
| Installer sub component: | openshift-ansible | QA Contact: | Johnny Liu <jialiu> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | high | ||
| Priority: | high | CC: | bleanhar, jialiu, jrosenta, oarribas, rheinzma, rteague, xtian |
| Version: | 3.11.0 | ||
| Target Milestone: | --- | ||
| Target Release: | 3.11.z | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: |
The underlying problem here is that the 3.10 version of oc does not have the --field-selector flag whereas the 3.11 version does. Since this task is being performed just prior to client upgrade, the task fails because it is using the 3.10 version.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-10-22 11:02:22 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
This needs to be fixed. Just to be clear, when you say "it's possible to see the error", the error happens every time, right? Hello Brenton, Yes, it's happening always like it's commented: "[How reproducible] Always" Dues the ansible change that was introduced in the version indicated in the description since "oc delete pods" doesn't have the option "--field-selector=". Then, it's very easy to verify. Take care, that "oc get pods" will work with the option "--field-selector=", but not the delete. The underlying problem here is that the 3.10 version of `oc` does not have the --field-selector flag whereas the 3.11 version does. Since this task is being performed just prior to client upgrade, the task fails because it is using the 3.10 version. Verified this bug with openshift-ansible-3.11.287-1.git.0.6837046.el7.noarch, and PASS. Before: [root@jialiu2310master-etcd-nfs-1 ~]# oc version oc v3.10.183 kubernetes v1.10.0+b81c8f8 features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://jialiu2310master-etcd-nfs-1.int.0917-omm.qe.rhcloud.com:8443 openshift v3.10.181 kubernetes v1.10.0+b81c8f8 Upgrading: 09-17 17:36:10 TASK [openshift_node : Delete OpenShift SDN/OVS pods prior to upgrade] ********* 09-17 17:36:11 changed: [ci-vm-10-0-148-44.hosted.upshift.rdu2.redhat.com -> ci-vm-10-0-148-44.hosted.upshift.rdu2.redhat.com] => { 09-17 17:36:11 "changed": true, 09-17 17:36:11 "cmd": "oc get pods --config=/etc/origin/master/admin.kubeconfig --field-selector=spec.nodeName=jialiu2310master-etcd-nfs-1 -o json -n openshift-sdn | oc delete --config=/etc/origin/master/admin.kubeconfig --force --grace-period=0 -f -\n", 09-17 17:36:11 "delta": "0:00:00.665724", 09-17 17:36:11 "end": "2020-09-17 05:36:11.705036", 09-17 17:36:11 "invocation": { 09-17 17:36:11 "module_args": { 09-17 17:36:11 "_raw_params": "oc get pods --config=/etc/origin/master/admin.kubeconfig --field-selector=spec.nodeName=jialiu2310master-etcd-nfs-1 -o json -n openshift-sdn | oc delete --config=/etc/origin/master/admin.kubeconfig --force --grace-period=0 -f -\n", 09-17 17:36:11 "_uses_shell": true, 09-17 17:36:11 "argv": null, 09-17 17:36:11 "chdir": null, 09-17 17:36:11 "creates": null, 09-17 17:36:11 "executable": null, 09-17 17:36:11 "removes": null, 09-17 17:36:11 "stdin": null, 09-17 17:36:11 "stdin_add_newline": true, 09-17 17:36:11 "strip_empty_ends": true, 09-17 17:36:11 "warn": true 09-17 17:36:11 } 09-17 17:36:11 }, 09-17 17:36:11 "rc": 0, 09-17 17:36:11 "start": "2020-09-17 05:36:11.039312", 09-17 17:36:11 "stderr": "warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.", 09-17 17:36:11 "stderr_lines": [ 09-17 17:36:11 "warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely." 09-17 17:36:11 ], 09-17 17:36:11 "stdout": "pod \"ovs-2kcr5\" force deleted\npod \"sdn-4g58s\" force deleted", 09-17 17:36:11 "stdout_lines": [ 09-17 17:36:11 "pod \"ovs-2kcr5\" force deleted", 09-17 17:36:11 "pod \"sdn-4g58s\" force deleted" 09-17 17:36:11 ] 09-17 17:36:11 } After: [root@jialiu2310master-etcd-nfs-1 ~]# oc version oc v3.11.287 kubernetes v1.11.0+d4cacc0 features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://jialiu2310master-etcd-nfs-1.int.0917-omm.qe.rhcloud.com:8443 openshift v3.11.287 kubernetes v1.11.0+d4cacc0 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 3.11.306 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4170 |
[Description of problem] Trying to upgrade from OCP 3.10.181 to OCP 3.11.z, it's possible to see the error ~~~ fatal: [master.example.com -> master.example.com]: FAILED! => { "changed": true, "cmd": "oc delete pods --config=/etc/origin/master/admin.kubeconfig --field-selector=spec.nodeName=master.example.com -n openshift-sdn --force --grace-period=0\n", "delta": "0:00:00.477759", "end": "2020-06-29 15:41:29.652942", "invocation": { "module_args": { "_raw_params": "oc delete pods --config=/etc/origin/master/admin.kubeconfig --field-selector=spec.nodeName=master.example.com -n openshift-sdn --force --grace-period=0\n", "_uses_shell": true, "argv": null, "chdir": null, "creates": null, "executable": null, "removes": null, "stdin": null, "stdin_add_newline": true, "strip_empty_ends": true, "warn": true } }, "msg": "non-zero return code", "rc": 1, "start": "2020-06-29 15:41:29.175183", "stderr": "Error: unknown flag: --field-selector\n\n\nUsage:\n oc delete ([-f FILENAME] | TYPE [(NAME | -l label | --all)]) [flags]\n\nExamples:\n # Delete a pod using the type and ID specified in pod.json.\n oc delete -f pod.json\n \n # Delete a pod based on the type and ID in the JSON passed into stdin.\n cat pod.json | oc delete -f -\n \n # Delete pods and services with label name=myLabel.\n oc delete pods,services -l name=myLabel\n \n # Delete a pod with name node-1-vsjnm.\n oc delete pod node-1-vsjnm\n \n # Delete all resources associated with a running app, includes\n # buildconfig,deploymentconfig,service,imagestream,route and pod,\n # where 'appName' is listed in 'Labels' of 'oc describe [resource] [resource name]' output.\n oc delete all -l app=appName\n \n # Delete all pods\n oc delete pods --all\n\nOptions:\n --all=false: Delete all resources, including uninitialized ones, in the namespace of the specified resource types.\n --cascade=true: If true, cascade the deletion of the resources managed by this resource (e.g. Pods created by a ReplicationController). Default true.\n -f, --filename=[]: Filename, directory, or URL to files containing the resource to delete.\n --force=false: Only used when grace-period=0. If true, immediately remove resources from API and bypass graceful deletion. Note that immediate deletion of some resources may result in inconsistency or data loss and requires confirmation.\n --grace-period=-1: Period of time in seconds given to the resource to terminate gracefully. Ignored if negative. Set to 1 for immediate shutdown. Can only be set to 0 when --force is true (force deletion).\n --ignore-not-found=false: Treat \"resource not found\" as a successful delete. Defaults to \"true\" when --all is specified.\n --include-extended-apis=true: If true, include definitions of new APIs via calls to the API server. [default true]\n --include-uninitialized=false: If true, the kubectl command applies to uninitialized objects. If explicitly set to false, this flag overrides other flags that make the kubectl commands apply to uninitialized objects, e.g., \"--all\". Objects with empty metadata.initializers are regarded as initialized.\n --now=false: If true, resources are signaled for immediate shutdown (same as --grace-period=1).\n -o, --output='': Output mode. Use \"-o name\" for shorter output (resource/name).\n -R, --recursive=false: Process the directory used in -f, --filename recursively. Useful when you want to manage related manifests organized within the same directory.\n -l, --selector='': Selector (label query) to filter on, not including uninitialized ones.\n --timeout=0s: The length of time to wait before giving up on a delete, zero means determine a timeout from the size of the object\n\nUse \"oc options\" for a list of global command-line options (applies to all commands).", "stderr_lines": [ "Error: unknown flag: --field-selector", "", ... Failure summary: 1. Hosts: master.example.com Play: Update master nodes Task: Delete OpenShift SDN/OVS pods prior to upgrade Message: non-zero return code ~~~ [Version-Release number of selected component (if applicable)] Upgrading from OCP 3.10.181 to OCP 3.11 [How reproducible] Always [Steps to Reproduce] 1. Follow the documentation [1] to upgrade from 3.10.181 to OCP 3.11 [Actual results] Error in the task " Delete OpenShift SDN/OVS pods prior to upgrade" when is run the command: "cmd": "oc delete pods --config=/etc/origin/master/admin.kubeconfig --field-selector=spec.nodeName=master.example.com -n openshift-sdn --force --grace-period=0\n" [Expected results] Upgrade finish with success [Additional info] This issue is dues to this commit [2] done in May 2020 wherein the upgrade task. It was changed the line: ~~~ $ oc get pods --config={{ openshift.common.config_base }}/master/admin.kubeconfig --field-selector=spec.nodeName={{ l_kubelet_node_name | lower }} ~~~ By: ~~~ $ oc delete pods --config={{ openshift.common.config_base }}/master/admin.kubeconfig --field-selector=spec.nodeName={{ l_kubelet_node_name | lower }} ~~~ The "oc get" has the field-selector implemented, but it's not for "oc delete pods" and this is the reason of the error. [WORKAROUND] The workaround describe below worked. Move the packages to the previous version where the commit above was not present, this version is the 3.11.216. For doing it, if the upgrade process was starting, it's needed to downgrade some packages like this: 1. Move the openshift-ansible t othe version 3.11.216 (two versions before the latest 3.11.232). Where that change in the code was introduced: ~~~ ### Check in the bastion host the openshift-ansible version that you have. It should be like this: # rpm -qa|grep openshift-ansible openshift-ansible-playbooks-3.11.232-1.git.0.f0f2213.el7.noarch openshift-ansible-docs-3.11.232-1.git.0.f0f2213.el7.noarch openshift-ansible-roles-3.11.232-1.git.0.f0f2213.el7.noarch openshift-ansible-3.11.232-1.git.0.f0f2213.el7.noarch ### Downgrade the packages to the 3.11.216 version # yum downgrade openshift-ansible-3.11.216-1.git.0.085486a.el7.noarch openshift-ansible-docs-3.11.216-1.git.0.085486a.el7.noarch openshift-ansible-playbooks-3.11.216-1.git.0.085486a.el7.noarch openshift-ansible-roles-3.11.216-1.git.0.085486a.el7.noarch # rpm -qa|grep openshift-ansible openshift-ansible-docs-3.11.216-1.git.0.085486a.el7.noarch openshift-ansible-playbooks-3.11.216-1.git.0.085486a.el7.noarch openshift-ansible-3.11.216-1.git.0.085486a.el7.noarch openshift-ansible-roles-3.11.216-1.git.0.085486a.el7.noarch # rpm -qa|grep openshift atomic-openshift-docker-excluder-3.11.232-1.git.0.a5bc32f.el7.noarch atomic-openshift-node-3.10.181-1.git.0.3ab4b3d.el7.x86_64 openshift-ansible-docs-3.11.216-1.git.0.085486a.el7.noarch atomic-openshift-excluder-3.11.232-1.git.0.a5bc32f.el7.noarch atomic-openshift-clients-3.10.181-1.git.0.3ab4b3d.el7.x86_64 atomic-openshift-hyperkube-3.10.181-1.git.0.3ab4b3d.el7.x86_64 openshift-ansible-playbooks-3.11.216-1.git.0.085486a.el7.noarch openshift-ansible-3.11.216-1.git.0.085486a.el7.noarch atomic-openshift-3.10.181-1.git.0.3ab4b3d.el7.x86_64 openshift-ansible-roles-3.11.216-1.git.0.085486a.el7.noarch 2. Check if you have more packages in the 232 version, in my case, at this moment, I have the atomic-openshift-docker-excluder, and I could downgrade them in the same way: ~~~ ### List other openshift packages using the 232 version # rpm -qa|grep openshift.*232 atomic-openshift-docker-excluder-3.11.232-1.git.0.a5bc32f.el7.noarch atomic-openshift-excluder-3.11.232-1.git.0.a5bc32f.el7.noarch ### List the name of the previous packages using the 216 version # yum search --show-duplicates atomic-openshift-docker-excluder atomic-openshift-excluder |grep 3.11.216 atomic-openshift-docker-excluder-3.11.216-1.git.0.198d885.el7.noarch : Exclude atomic-openshift-excluder-3.11.216-1.git.0.198d885.el7.noarch : Exclude ### Downgrade these packages # yum downgrade atomic-openshift-docker-excluder-3.11.216-1.git.0.198d885.el7.noarch atomic-openshift-excluder-3.11.216-1.git.0.198d885.el7.noarch 3. Check the openshift packages in the nodes. You could want to use ansible to see the versions that you have in the nodes. They shouldn't have any package with version 3.11.232, but they could have some packages in version 3.10.x ~~~ # rpm -qa|grep openshift atomic-openshift-node-3.10.181-1.git.0.3ab4b3d.el7.x86_64 openshift-ansible-docs-3.11.216-1.git.0.085486a.el7.noarch atomic-openshift-clients-3.10.181-1.git.0.3ab4b3d.el7.x86_64 atomic-openshift-hyperkube-3.10.181-1.git.0.3ab4b3d.el7.x86_64 openshift-ansible-playbooks-3.11.216-1.git.0.085486a.el7.noarch openshift-ansible-3.11.216-1.git.0.085486a.el7.noarch atomic-openshift-docker-excluder-3.11.216-1.git.0.198d885.el7.noarch atomic-openshift-3.10.181-1.git.0.3ab4b3d.el7.x86_64 openshift-ansible-roles-3.11.216-1.git.0.085486a.el7.noarch atomic-openshift-excluder-3.11.216-1.git.0.198d885.el7.noarch ~~~ 4. Modify the inventory file to fix the version to 3.11.216 adding: ~~~ [OSEv3:vars] ... openshift_pkg_version=-3.11.216 openshift_image_tag=v3.11.216 ... ~~~ 5. Run the upgrade playbook ~~~ $ cd /usr/share/ansible/openshift-ansible $ ansible-playbook -i <path to inventory file> playbooks/byo/openshift-cluster/upgrades/v3_11/upgrade_control_plane.yml ~~~ 6. Now, it's possible to continue with the upgrade like always When the previous upgrade is finished with success, it's possible to move to the latest OCP 3.11.z [1] https://docs.openshift.com/container-platform/3.11/upgrading/automated_upgrades.html#upgrading-ocp [2] https://github.com/openshift/openshift-ansible/commit/a69085bff1ac0482d7f1437ec50e2798363d17df