Bug 1320557

Description of problem:
When unregistering a content host (either by subscription-manager unregister, or deleting Content Host, or deleting Host), Actions::Katello::System::Destroy task is being processed.

This task has a concurrency bug that with some probability causes Actions::Candlepin::ListenOnCandlepinEvents task to be paused with error:

Katello::Resources::Candlepin::Consumer: 410 Gone {"displayMessage":"Unit b3012a90-41b6-4788-98a9-5b41839b6dca has been deleted","requestUuid":"04dbfeda-75cb-4ffa-ad9f-9f4c818fc868","deletedId":"b3012a90-41b6-4788-98a9-5b41839b6dca"} (GET /candlepin/consumers/b3012a90-41b6-4788-98a9-5b41839b6dca)

(UUID matches the UUID of just being deleted Content Host)

sequence of steps leading to the bug:
- Actions::Candlepin::Consumer::Destroy sub-task executed
  - it deletes the consumer from candlepin and recalculates compliance for it (imho redundant step when we delete the consumer, but katello is aware of it)
  - it announces the compliance.create event to ListenOnCandlepinEvents task
  - katello finds the katello_system is present so it runs "re-indexing content host .."
- the parent task Actions::Katello::System::Destroy even _now_ enters Actions::Katello::System::Destroy
  - this subtask deletes the system from katello_systems _after_ the check before "re-indexing content host .." is made, so the re-index is _not_ skipped
- the re-index of the content host calls GET consumer/<uuid> on candlepin, what triggers the 410 Gone in ListenOnCandlepinEvents task


Version-Release number of selected component (if applicable):
Sat 6.1.7


How reproducible:
100% within few minutes


Steps to Reproduce:
1. tail -f /var/log/foreman/production.log | grep -e "in phase Finalize Actions::Katello::System::Destroy" -e "skip re-indexing of non-existent content host" -e "re-indexing content host"

2. Have opened Actions::Candlepin::ListenOnCandlepinEvents task in WebUI

3. On some Content Host, register and unregister it in a loop (here RHEL7 used, update if using RHEL5 or 6 accordingly):

while true; do
  subscription-manager register --force --org="Default_Organization" --environment="Library" --username=admin --password=faYakexMm5XN543x
  subscription-manager subscribe --pool=8aa2d415526494380152732fc8d20dd7
  subscription-manager repos --enable rhel-7-server-rpms --enable rhel-7-server-satellite-tools-6.1-rpms
  date
  subscription-manager unregister
  date
  sleep 5
done

4. monitor the tail -f output and ListenOnCandlepinEvents task


Actual results:
- tail shows:

2016-03-23 14:17:05 [D] re-indexing content host pmoravec-rhel7.gsslab.brq.redhat.com
2016-03-23 14:17:05 [D]          Step f5705893-2577-41c4-9af6-9b7c10ccb646: 6   running >>   success in phase Finalize Actions::Katello::System::Destroy

- ListenOnCandlepinEvents task is paused/error with error:

Katello::Resources::Candlepin::Consumer: 410 Gone {"displayMessage":"Unit b3012a90-41b6-4788-98a9-5b41839b6dca has been deleted","requestUuid":"04dbfeda-75cb-4ffa-ad9f-9f4c818fc868","deletedId":"b3012a90-41b6-4788-98a9-5b41839b6dca"} (GET /candlepin/consumers/b3012a90-41b6-4788-98a9-5b41839b6dca)


Expected results:
- tail can be ok (rather symptom for devels), just the ListenOnCandlepinEvents task needs to be running without an error


Additional info:
seems like a lack of locking / concurrency bug where reindex_pool_subscription_handler.rb:

      def reindex_consumer(message)
        if message.content['newEntity']
          uuid = JSON.parse(message.content['newEntity'])['consumer']['uuid']
          system = ::Katello::System.find_by_uuid(uuid)
          if system.nil?
            @logger.debug "skip re-indexing of non-existent content host #{uuid}"
          else
            @logger.debug "re-indexing content host #{system.name}"
            system.update_index
          end

needs to be executed as atomic operation (not concurrently with Actions::Katello::System::Destroy)