Bug 1409573 - Race conditions during simultaneous manipulations with capsules organizations (create/delete)
Summary: Race conditions during simultaneous manipulations with capsules organizations...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Foreman Proxy
Version: 6.2.6
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: Roman Plevka
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-01-02 13:53 UTC by Roman Plevka
Modified: 2018-08-02 20:59 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-08-02 20:59:11 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Roman Plevka 2017-01-02 13:53:11 UTC
Description of problem:
There's a race condition during creating a new smart proxy (capsule) and removing any org at (almost) the same time resulting in a ISE for the proxy create command (request):

# hammer -v -u admin -p changeme --output=csv proxy create --url="https://sat6.com:13249" --name="foo1"
[ERROR 2017-01-02 08:41:29 Exception] ERROR:  insert or update on table "taxable_taxonomies" violates foreign key constraint "taxable_taxonomies_taxonomy_id_fk"
DETAIL:  Key (taxonomy_id)=(9) is not present in table "taxonomies".

Could not create the proxy:
  ERROR:  insert or update on table "taxable_taxonomies" violates foreign key constraint "taxable_taxonomies_taxonomy_id_fk"
  DETAIL:  Key (taxonomy_id)=(9) is not present in table "taxonomies".
 
Looks like there's a query for existing orgs (as the newly created capsule is being assigned to all organizations by default). However, with right timing, if user issues Org delete operation just in the middle of the transaction, the whole operation fails.


Version-Release number of selected component (if applicable):
# rpm -qa satellite
satellite-6.2.6-2.0.el7sat.noarch

How reproducible:
timing-sensitive but possible

Steps to Reproduce:
1. have a spare capsule to add to the satellite (possibly create a fake proxy by forwarding the 9090 port)
2. have a spare organization created.
3. execute the proxy create and org delete commands at the same time (i use robottelo for the org delete part, so it takes more time to take effect (robottelo needs to ssh to the box and execute the command) so I execute the Org delete command a second later).
4. eventually, the proxy create command fails with the above error message

- possibly create a loop that would create and delete both entities over and over again and you should be able to hit it.

Actual results:


Expected results:


Additional info:

Comment 2 Roman Plevka 2017-04-20 12:15:11 UTC
hit this again

hammer -v -u admin -p changeme --output=csv proxy create --url="https://sat6.com:11676" --name="fSIFbn6ees"
2017-04-19 20:18:35 - robottelo.ssh - INFO - <<< stderr
[ERROR 2017-04-19 20:18:35 Exception] ERROR:  insert or update on table "katello_capsule_lifecycle_environments" violates foreign key constraint "katello_capsule_lifecycle_environments_environment_fk"
DETAIL:  Key (lifecycle_environment_id)=(346) is not present in table "katello_environments".

Could not create the proxy:
  ERROR:  insert or update on table "katello_capsule_lifecycle_environments" violates foreign key constraint "katello_capsule_lifecycle_environments_environment_fk"
  DETAIL:  Key (lifecycle_environment_id)=(346) is not present in table "katello_environments".

Comment 3 Roman Plevka 2017-04-28 10:01:09 UTC
hit this again  (on removing location)

2017-04-27 00:59:00 - nailgun.client - DEBUG - Making HTTP DELETE request to https://sat6.com/api/v2/locations/240 with options {'verify': False, 'auth': ('admin', 'changeme'), 'headers': {'content-type': 'application/json'}} and no data.

2017-04-27 00:59:04 - nailgun.client - WARNING - Received HTTP 500 response: {
  "error": {"message":"ERROR:  update or delete on table \"taxonomies\" violates foreign key constraint \"taxable_taxonomies_taxonomy_id_fk\" on table \"taxable_taxonomies\"\nDETAIL:  Key (id)=(240) is still referenced from table \"taxable_taxonomies\".\n"}
}

Comment 4 Roman Plevka 2017-05-05 08:39:35 UTC
Another table, where RC happens is "katello_capsule_lifecycle_environments"
this time this occurred on removing a capsule and adding organizations.

2016-12-21 05:49:00 [app] [W] Action failed
 | PG::Error: ERROR:  update or delete on table "smart_proxies" violates foreign key constraint "katello_capsule_lifecycle_environments_capsule_fk" on table "katello_capsule_lifecycle_environments"
 | DETAIL:  Key (id)=(7) is still referenced from table "katello_capsule_lifecycle_environments".


It most probably collided with the following request:

2016-12-21 05:48:57 [app] [I] Started POST "/katello/api/organizations/846/environments" for ::1 at 2016-12-21 05:48:57 +0100
2016-12-21 05:48:57 [app] [I]   Rendered api/v2/taxonomies/show.json.rabl (231.7ms)
2016-12-21 05:48:57 [app] [I] Completed 200 OK in 263ms (Views: 218.8ms | ActiveRecord: 19.4ms)
2016-12-21 05:48:57 [app] [I] Processing by Katello::Api::V2::EnvironmentsController#create as JSON
2016-12-21 05:48:57 [app] [I]   Parameters: {"name"=>"s4HW6z", "prior_id"=>721, "api_version"=>"v2", "organization_id"=>"846", "environment"=>{"name"=>"s4HW6z", "prior_id"=>721}}
2016-12-21 05:48:58 [app] [I] Authorized user admin(Admin User)
2016-12-21 05:48:58 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.90/app/views/katello/api/v2/common/create.json.rabl within katello/api/v2/layouts/resource (19.0ms)
2016-12-21 05:48:58 [app] [I] Completed 200 OK in 99ms (Views: 17.7ms | ActiveRecord: 14.4ms)
2016-12-21 05:48:58 [app] [I] Started POST "/katello/api/v2/repositories" for 10.8.0.99 at 2016-12-21 05:48:58 +0100
...

Comment 5 Bryan Kearney 2018-08-02 20:59:11 UTC
Thank you for your interest in Satellite 6. We have evaluated this request, and we do not expect this to be implemented in the product in the forseeable future. We are therefore closing this out as WONTFIX. If you have any concerns about this, please feel free to contact Rich Jerrido or Bryan Kearney. Thank you.


Note You need to log in before you can comment on or make changes to this bug.