Bug 1601338 - apb update failed at 'Set postgresql objects state=present'
Summary: apb update failed at 'Set postgresql objects state=present'
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Service Broker
Version: 3.10.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 3.11.0
Assignee: David Zager
QA Contact: Zihan Tang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-07-16 06:47 UTC by Zihan Tang
Modified: 2018-10-11 07:22 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
undefined
Clone Of:
Environment:
Last Closed: 2018-10-11 07:21:41 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:2652 0 None None None 2018-10-11 07:22:03 UTC

Description Zihan Tang 2018-07-16 06:47:10 UTC
Description of problem:
when update apb instance, the update failed at 'Set postgresql objects state=present ' with  error:

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: KeyError: 'rolling_params'
failed: [localhost] (item={u'state': u'present', u'name': u'deployment.yaml'}) => {"changed": false, "item": {"name": "deployment.yaml", "state": "present"}, "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_6Z6sL4/ansible_module_openshift_raw.py\", line 204, in <module>\n    main()\n  File \"/tmp/ansible_6Z6sL4/ansible_module_openshift_raw.py\", line 200, in main\n    OpenShiftRawModule().execute_module()\n  File \"/tmp/ansible_6Z6sL4/ansible_modlib.zip/ansible/module_utils/k8s/raw.py\", line 142, in execute_module\n  File \"/tmp/ansible_6Z6sL4/ansible_modlib.zip/ansible/module_utils/k8s/helper.py\", line 136, in object_from_params\n  File \"/tmp/ansible_6Z6sL4/ansible_modlib.zip/ansible/module_utils/k8s/helper.py\", line 301, in __set_obj_attribute\n  File \"/tmp/ansible_6Z6sL4/ansible_modlib.zip/ansible/module_utils/k8s/helper.py\", line 301, in __set_obj_attribute\n  File \"/tmp/ansible_6Z6sL4/ansible_modlib.zip/ansible/module_utils/k8s/helper.py\", line 271, in __set_obj_attribute\nKeyError: 'rolling_params'\n", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 1}


Version-Release number of selected component (if applicable):
asb: 1.2.17
apb: v3.10.18

How reproducible:
random, about 2 times in OCP 3.10 testing

Steps to Reproduce:
1.provision postgresql-apb
2.update the plan 
3. check the update sandbox

Actual results:
the update pod failed with error:

[root@preserve-share31017ceph-master-etcd-1 ~]# oc get pod 
NAME                                       READY     STATUS    RESTARTS   AGE
apb-54fd2418-10e5-45b1-8362-97f12bbf933f   0/1       Error     0          2d
[root@preserve-share31017ceph-master-etcd-1 ~]# oc logs -f apb-54fd2418-10e5-45b1-8362-97f12bbf933f

PLAY [postgresql-apb update] ***************************************************

TASK [ansibleplaybookbundle.asb-modules : debug] *******************************
skipping: [localhost]

TASK [postgresql-apb : Update last operation] **********************************
changed: [localhost]

TASK [postgresql-apb : Determine cluster type] *********************************
changed: [localhost]

TASK [postgresql-apb : Set facts] **********************************************
ok: [localhost]

TASK [postgresql-apb : Include variables based on ansible version] *************
skipping: [localhost]

TASK [postgresql-apb : Backup database] ****************************************
changed: [localhost]

TASK [postgresql-apb : Rsync to APB] *******************************************
changed: [localhost]

TASK [postgresql-apb : Take down old deployment and pvc (if it exists)] ********
changed: [localhost] => (item=pvc.yaml)
changed: [localhost] => (item=deployment.yaml)

TASK [postgresql-apb : Update last operation] **********************************
changed: [localhost]

TASK [postgresql-apb : Set postgresql objects state=present] *******************
ok: [localhost] => (item={u'state': u'present', u'name': u'service.yaml'})
ok: [localhost] => (item={u'state': u'absent', u'name': u'pvc.yaml'})
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: KeyError: 'rolling_params'
failed: [localhost] (item={u'state': u'present', u'name': u'deployment.yaml'}) => {"changed": false, "item": {"name": "deployment.yaml", "state": "present"}, "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_6Z6sL4/ansible_module_openshift_raw.py\", line 204, in <module>\n    main()\n  File \"/tmp/ansible_6Z6sL4/ansible_module_openshift_raw.py\", line 200, in main\n    OpenShiftRawModule().execute_module()\n  File \"/tmp/ansible_6Z6sL4/ansible_modlib.zip/ansible/module_utils/k8s/raw.py\", line 142, in execute_module\n  File \"/tmp/ansible_6Z6sL4/ansible_modlib.zip/ansible/module_utils/k8s/helper.py\", line 136, in object_from_params\n  File \"/tmp/ansible_6Z6sL4/ansible_modlib.zip/ansible/module_utils/k8s/helper.py\", line 301, in __set_obj_attribute\n  File \"/tmp/ansible_6Z6sL4/ansible_modlib.zip/ansible/module_utils/k8s/helper.py\", line 301, in __set_obj_attribute\n  File \"/tmp/ansible_6Z6sL4/ansible_modlib.zip/ansible/module_utils/k8s/helper.py\", line 271, in __set_obj_attribute\nKeyError: 'rolling_params'\n", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 1}

PLAY RECAP *********************************************************************
localhost                  : ok=7    changed=6    unreachable=0    failed=1   

Expected results:
update should succeed.

Additional info:

Comment 1 Zihan Tang 2018-07-16 06:49:33 UTC
The bug is very random reproduced in v3.10. if it is hard to find the root cause, just want to use it to trace this issue.

Comment 2 David Zager 2018-08-06 19:03:24 UTC
For 3.11 we have moved to the `k8s` module in Ansible 2.6 with the dynamic python client. I believe that these issues should be addressed by this change.

Comment 4 Zihan Tang 2018-08-20 07:04:10 UTC
It's a rarely reproduced bug, 
We haven't hit it in the 3.11 run,  mark it as verified. 
if we reproduce again, will re-open it.

ASB version : v1.3.9

Comment 6 errata-xmlrpc 2018-10-11 07:21:41 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2652


Note You need to log in before you can comment on or make changes to this bug.