Bug 1511362

Summary: satellite-clone migrating Sat6 from RHEL6 causes Sync Plans are run twice
Product: Red Hat Satellite Reporter: Pavel Moravec <pmoravec>
Component: Satellite CloneAssignee: John Mitsch <jomitsch>
Status: CLOSED ERRATA QA Contact: sthirugn <sthirugn>
Severity: medium Docs Contact:
Priority: high    
Version: 6.2.12CC: bbuckingham, sthirugn
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-02-19 17:23:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Pavel Moravec 2017-11-09 08:57:39 UTC
Description of problem:
When one clones a Satellite6 from RHEL6 via satellite-clone, any Sync Plan (either cloned from a parent Satellite or a new one created) is attempted to execute twice on every schedule.

This problem does _not_ happen when cloning Sat6 on RHEL7. Only when cloning from RHEL6 (where katello-backup --online-backup was run)


The root cause is the following:
- satellite-clone installs Satellite incl. filling event_listeners mongo collection
- this collection is not dropped (cf. [1]) while I guess it should be
- once the satellite-clone finishes, event_listeners collection in mongo has 2 records (original from installer and one copied from cloned Sat)
- how Sync Plan is triggered:
  - pulp checks scheduled_calls collection to trigger repo synces
  - once a repo sync finishes, hooks from event_listeners are run to notify katello/foreman
  - so katello gets such trigger and files foreman-task for repo sync (with pulp task ID such that the foreman-task wont trigger new pulp sync task, but just checks it status)
  - pulp executes the triggers for all applicable event_listeners, so having a duplicate ones, a duplicate foreman-task is created (for each and every repo synced)


[1] https://github.com/RedHatSatellite/satellite-clone/blob/master/roles/satellite-clone/tasks/reset_pulp_data.yml#L2


Version-Release number of selected component (if applicable):
satellite-clone-1.1.4-1.el7sat.noarch


How reproducible:
100%


Steps to Reproduce:
1. Clone whatever Satellite (even freshly installed)
2. Have (cloned or created) Sync Plan (idealy with hourly frequency to see it soon)
3. Wait for the Sync Plan execution time and check foreman tasks created
4. check event listeners:
mongo pulp_database --eval "DBQuery.shellBatchSize = 10000000; db.event_listeners.find().shellPrint()"


Actual results:
3. Shows duplicate tasks created / half of them failing with "acquired lock .." error.
4. shows 2 listeners


Expected results:
3. no extra tasks created
4. to show just 1 listener

Additional info:
I was monitoring content of the events_listener table. The table got one new record, then:
- when cloning from RHEL7, this record was removed and replaced by the same record from original Satellite
- when cloning from RHEL6, the record from original Satellite was just _added_ but previous record remained there - thats the problem

(just out of curiosity (if I am right with the way of fix): why we dont drop *all* tables but just some of them?)

Comment 3 John Mitsch 2017-11-30 18:24:30 UTC
PR is open upstream https://github.com/RedHatSatellite/satellite-clone/pull/266

Comment 4 sthirugn@redhat.com 2018-01-15 20:36:20 UTC
Upstream PR merged

Comment 6 sthirugn@redhat.com 2018-02-06 21:17:17 UTC
Verified in satellite-clone-1.2.1-1.el7sat.noarch

Post clone, the sync tasks were not duplicated.

Also only one listener is found:
# mongo pulp_database --eval "DBQuery.shellBatchSize = 10000000; db.event_listeners.find().shellPrint()"
MongoDB shell version: 2.6.11
connecting to: pulp_database
{ "_id" : ObjectId("59d51412f3a8852ba0948105"), "notifier_config" : { "url" : "https://dhcp182-245.gsslab.rdu2.redhat.com/katello/api/v2/repositories/sync_complete?token=JXc9L4QtiiPTFxv8nmi4PW3Rq9j9qTju" }, "_ns" : "event_listeners", "event_types" : [ "repo.sync.finish" ], "id" : "59d51412f3a8852ba0948105", "notifier_type_id" : "http" }

Comment 9 errata-xmlrpc 2018-02-19 17:23:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0330