Bug 1879587

Summary: simultaneous POST requests to some endpoints result in Different response payload format
Product: Red Hat Satellite Reporter: Roman Plevka <rplevka>
Component: APIAssignee: Eric Helms <ehelms>
Status: CLOSED ERRATA QA Contact: Lukáš Hellebrandt <lhellebr>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.8.0CC: apatel, gtalreja, lzap, pcreech, rabajaj, tbrisker
Target Milestone: 6.9.0Keywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: foreman-2.3.0-1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-04-21 13:17:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Roman Plevka 2020-09-16 14:58:06 UTC
Description of problem:

Whenever I run multiple POST requests to create discovery rules in parralel, I ocassionally get a 201 response with a different payload format.
I’m unable to reproduce this if I execute the requests sequentially. More parralel requests I execute, bigger change of hitting this issue:

{"discovery_rule":{"id":742,"name":"57531206","search":"foo","hostgroup_id":125,"hostname":"foo","max_count":0,"priority":29227,"enabled":true,"created_at":"2020-09-16T10:33:11.220Z","updated_at":"2020-09-16T10:33:11.220Z"}}
{"id":745,"name":"304796992","search":"foo","hostgroup_id":125,"hostname":"foo","max_count":0,"priority":17120,"enabled":true,"created_at":"2020-09-16T10:33:12.128Z","updated_at":"2020-09-16T10:33:12.128Z"}


I provided logs and reproducer here:
https://community.theforeman.org/t/different-response-format-on-identical-requests-when-executed-in-parralel/20511


Version-Release number of selected component (if applicable):
6.8.0-15

How reproducible:
on concurrent POST requests to /discovery_rules

Steps to Reproduce:
1. create an org, location and a hostgroup which you want to associate your rules with
2. run the following example in multiple shells (3-4):

for i in {1..10}; do curl -sku admin:changeme -H "content-type: application/json" -d "{\"name\": \"$RANDOM$RANDOM\", \"hostname\": \"myhost-<%= rand(99999) %>\", \"priority\": $RANDOM, \"hostgroup_id\": 125, \"location_ids\": [259], \"organization_ids\": [255], \"search\": \"memory > 500\"}" https://<sat_server>/api/v2/discovery_rules; echo ""; done

Actual results:
some responses use different payload format:

{"id":2969,"name":"93955553","search":"foo","hostgroup_id":125,"hostname":"foo","max_count":0,"priority":18417,"enabled":true,"created_at":"2020-09-16T14:51:28.870Z","updated_at":"2020-09-16T14:51:28.870Z"}
{"id":2972,"name":"102037951","search":"foo","hostgroup_id":125,"hostname":"foo","max_count":0,"priority":10731,"enabled":true,"created_at":"2020-09-16T14:51:29.796Z","updated_at":"2020-09-16T14:51:29.796Z"}
{"discovery_rule":{"id":2975,"name":"103111213","search":"foo","hostgroup_id":125,"hostname":"foo","max_count":0,"priority":16633,"enabled":true,"created_at":"2020-09-16T14:51:30.869Z","updated_at":"2020-09-16T14:51:30.869Z"}}
^^^
{"id":2977,"name":"179169626","search":"foo","hostgroup_id":125,"hostname":"foo","max_count":0,"priority":15149,"enabled":true,"created_at":"2020-09-16T14:51:31.881Z","updated_at":"2020-09-16T14:51:31.881Z"}


Expected results:
ALL responses should use consistent format

Comment 1 Roman Plevka 2020-09-17 12:50:00 UTC
I was able to reproduce this on some different endpoints. Looks like this is not related to Discovery component but rather API.


/api/v2/roles/clone
/api/v2/discovered_hosts/facts
/api/v2/discovery_rules/


It's insteresting that this is not reproducible on /api/v2/roles endpoint but it is on /api/v2/roles/clone.

Comment 4 Tomer Brisker 2020-09-17 14:50:39 UTC
Copying my comment from discourse: 

That’s quite odd, the response content should be generated by the foreman application, regardless of which webserver it is running under.
However, I did find one thing in common for all three endpoints identified: They don’t have matching view templates.
I wounder if copying e.g. <$discovery_plugin>/app/views/api/v2/discovery_rules/show.json.rabl to <$discovery_plugin>/app/views/api/v2/discovery_rules/create.json.rabl would lead to any difference?

Comment 7 Tomer Brisker 2021-01-07 08:27:20 UTC
Connecting redmine issue https://projects.theforeman.org/issues/30946 from this bug

Comment 9 Lukáš Hellebrandt 2021-02-11 15:37:18 UTC
Verified with Sat 6.9 snap 12.

Used reproducer from OP in 6 simultaneous processes, each of them doing 1000 calls. All of the responses were formed like this:

{"name":"2392011492","enabled":true,"hostgroup_id":1,"hostgroup_name":"testgrp","hostname":"myhost-<%= rand(99999) %>","priority":26362,"search":"memory > 500","hosts_limit":0,"id":5794,"hosts":[],"organizations":[{"id":1,"name":"Default Organization","title":"Default Organization","description":null}],"locations":[{"id":2,"name":"Default Location","title":"Default Location","description":null}]}

None of them had top-level entity key.

Comment 12 errata-xmlrpc 2021-04-21 13:17:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Satellite 6.9 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:1313