Bug 1595324

Summary: Cloudforms Automation not executing properly when multiple pods are created or killed in a short timeframe.
Product: Red Hat CloudForms Management Engine Reporter: Satoe Imaishi <simaishi>
Component: ProvidersAssignee: Beni Paskin-Cherniavsky <cben>
Status: CLOSED ERRATA QA Contact: juwatts
Severity: urgent Docs Contact:
Priority: urgent    
Version: 5.9.0CC: brahmani, cben, cpelland, gblomqui, jfrey, jhardy, jprause, mkanoor, mrobson, obarenbo, oourfali, smallamp, tfitzger
Target Milestone: GAKeywords: ZStream
Target Release: 5.9.3   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: 5.9.3.3 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1583832 Environment:
Last Closed: 2018-07-12 13:17:06 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: Container Management Target Upstream Version:
Embargoed:
Bug Depends On: 1583832    
Bug Blocks:    

Comment 2 Beni Paskin-Cherniavsky 2018-06-27 10:17:19 UTC
Tested on gaprindashvili, https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/264 ready for backport.

Comment 3 CFME Bot 2018-06-27 12:33:48 UTC
New commit detected on ManageIQ/manageiq-providers-kubernetes/gaprindashvili:

https://github.com/ManageIQ/manageiq-providers-kubernetes/commit/1748f9b993cf67fb053ec34673fd5080302a144f
commit 1748f9b993cf67fb053ec34673fd5080302a144f
Author:     Mooli Tayer <mtayer>
AuthorDate: Mon Jun 25 04:46:53 2018 -0400
Commit:     Mooli Tayer <mtayer>
CommitDate: Mon Jun 25 04:46:53 2018 -0400

    Merge pull request #264 from cben/event-ems-ref

    Set EmsEvent ems_ref to event's uid, to avoid same-second collision
    (cherry picked from commit 9ee61185a2c739f950e8021f7dd61578199ae350)

    Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1595324

 app/models/manageiq/providers/kubernetes/container_manager/event_catcher_mixin.rb | 3 +-
 app/models/manageiq/providers/kubernetes/container_manager/event_parser_mixin.rb | 3 +-
 spec/models/manageiq/providers/kubernetes/container_manager/event_catcher_mixin_spec.rb | 9 +-
 3 files changed, 10 insertions(+), 5 deletions(-)

Comment 5 brahmani 2018-07-02 08:40:30 UTC
Verified on CFME 5.9.3.3.20180627151410_682cecf.

verification instructions are on :  
https://bugzilla.redhat.com/show_bug.cgi?id=1583832#c20

After scale up 3 more redis pods, I get 3 POD_SCHEDULED events with same timestamp:

[root@dhcp-8-198-116 log]# grep '@@@@@' automation.log
[----] I, [2018-07-02T04:00:22.786675 #11914:fce630]  INFO -- : <AEMethod create_start_date_msg> @@@@@@@@@@@ container event: POD_SCHEDULED, timestamp: 2018-07-02 08:00:13 UTC, event uid: f30d3de9-7dcd-11e8-be2f-001a4a169854, project: kube-service-catalog, name: redis-1-d2qhv, message: Successfully assigned redis-1-d2qhv to benny-ocp-node2.qa.lab.tlv.redhat.com
[----] I, [2018-07-02T04:00:23.199953 #11923:fce630]  INFO -- : <AEMethod create_start_date_msg> @@@@@@@@@@@ container event: POD_SCHEDULED, timestamp: 2018-07-02 08:00:13 UTC, event uid: f31127fb-7dcd-11e8-be2f-001a4a169854, project: kube-service-catalog, name: redis-1-sfhtq, message: Successfully assigned redis-1-sfhtq to benny-ocp-node2.qa.lab.tlv.redhat.com
[----] I, [2018-07-02T04:00:23.321732 #11906:fce630]  INFO -- : <AEMethod create_start_date_msg> @@@@@@@@@@@ container event: POD_SCHEDULED, timestamp: 2018-07-02 08:00:13 UTC, event uid: f30ff4c4-7dcd-11e8-be2f-001a4a169854, project: kube-service-catalog, name: redis-1-q6c7s, message: Successfully assigned redis-1-q6c7s to benny-ocp-node3.qa.lab.tlv.redhat.com

Comment 7 errata-xmlrpc 2018-07-12 13:17:06 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2184