Bug 1271500

Description of problem:
Once Katello and Candlepin are out of sync it is for a user impossible to recover without deleting entries manual in the postgres database.

The Candlepin and Katello can get out of sync in the following situations
- Unexpected reboot
- Unexpected application restart during a subscription task
- Any other failing step in a Tasks that worked with Subscriptions

After such situation happends the Content Hosts page cannot be loaded:

2015-10-13 13:50:26 [I] Processing by Katello::Api::V2::SystemsController#index as */*
2015-10-13 13:50:26 [I]   Parameters: {"per_page"=>9999, "organization_id"=>3, "api_version"=>"v2", "system"=>{}}
2015-10-13 13:50:27 [I] Authorized user hoici(hoici )
2015-10-13 13:50:45 [I]   Rendered /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.67/app/views/katello/api/v2/systems/index.json.rabl within katello
/api/v2/layouts/collection (17945.0ms)
2015-10-13 13:50:46 [E] exception when talking to a remote client: Katello::Resources::Candlepin::Consumer: 410 Gone {"displayMessage":"Unit ed149cbb-9dd9-4
dd4-acb2-56ac6393b872 has been deleted","requestUuid":"17eb9c0d-7765-4cdf-b3d8-4423313d0904","deletedId":"ed149cbb-9dd9-4dd4-acb2-56ac6393b872"} (GET /candl
epin/consumers/ed149cbb-9dd9-4dd4-acb2-56ac6393b872) RestClient::Gone: Katello::Resources::Candlepin::Consumer: 410 Gone {"displayMessage":"Unit ed149cbb-9d
d9-4dd4-acb2-56ac6393b872 has been deleted","requestUuid":"17eb9c0d-7765-4cdf-b3d8-4423313d0904","deletedId":"ed149cbb-9dd9-4dd4-acb2-56ac6393b872"} (GET /c
andlepin/consumers/ed149cbb-9dd9-4dd4-acb2-56ac6393b872)
Body: {"displayMessage":"Unit ed149cbb-9dd9-4dd4-acb2-56ac6393b872 has been deleted","requestUuid":"17eb9c0d-7765-4cdf-b3d8-4423313d0904","deletedId":"ed149
cbb-9dd9-4dd4-acb2-56ac6393b872"}


Even the katello:reindex does not work to try to correct this:

# foreman-rake katello:reindex
API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
Elasticsearch Indices cleared.
Re-indexing Katello::ContentViewErratumFilterRule
Re-indexing Katello::HostCollection
Re-indexing Katello::System
Re-indexing Katello::ContentViewHistory
Re-indexing Katello::ContentViewFilter
Re-indexing Katello::TaskStatus
Re-indexing Katello::Job
Re-indexing Katello::ContentViewPuppetModule
Re-indexing Katello::Repository
Re-indexing Katello::ContentView
Re-indexing Katello::Distributor
Re-indexing Katello::ActivationKey
Re-indexing Katello::Provider
Re-indexing Katello::ContentViewPackageGroupFilterRule
Re-indexing Katello::ContentViewPackageFilterRule
Re-indexing Katello::Product
Re-indexing Katello::ContentViewPuppetEnvironment
Re-indexing Katello::Notice
Re-indexing Katello::Hypervisor
The following Katello::Hypervisor items could not be indexed due to various reasons.
Please check /usr/share/foreman/log/reindex.log for more detailed information.
Object: #<Katello::Hypervisor id: 9, uuid: "ed149cbb-9dd9-4dd4-acb2-56ac6393b872", name: "li-hc-1005", description: "Initial Registration Params", location: "None", environment_id: 2, created_at: "2015-09-21 10:41:02", updated_at: "2015-09-29 14:37:05", type: "Katello::Hypervisor", content_view_id: 52, host_id: nil>
rake aborted!
Hypervisor does not support this action

Tasks: TOP => katello:reindex


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Simulate an unexpected reboot, e.g. Delete a subscription in Candlepin without deleting it in Katello
2.
3.

Actual results:
Unable to recover without manual deleting a record in the katello database

Expected results:
Katello recovers, possible scenarios:
- It mentions that the COntent Host is Deleted
- Delete the ContentHost implicitly
Katello reindex succeeds
- Deletes the out of sync content host record from Katello


Additional info: