Bug 1336007
| Summary: | Postgresql ShareLock problems while configuring puppet concurrently | ||
|---|---|---|---|
| Product: | Red Hat Satellite | Reporter: | Pradeep Kumar Surisetty <psuriset> |
| Component: | Infrastructure | Assignee: | Chris Duryee <cduryee> |
| Status: | CLOSED ERRATA | QA Contact: | jcallaha |
| Severity: | urgent | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 6.2.0 | CC: | adprice, cwelton, dcaplan, jcallaha, jhutar, lzap, mmccune |
| Target Milestone: | Unspecified | Keywords: | PrioBumpPM, Triaged |
| Target Release: | Unused | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | foreman-1.11.0.53-1 | Doc Type: | Bug Fix |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-09-14 21:00:38 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Upstream bug component is Infrastructure Moving to POST since upstream bug http://projects.theforeman.org/issues/14076 has been closed Verified in Satellite 6.2.2 Snap 1.1
Performed the setup using docker content hosts.
First test was executed in serial.
[root@rhsm-qe-1 ~]# for i in {1..50}; do hammer -u admin -p changeme host update --name "test-puppet$i" --puppet-proxy "rhsm-qe-1.rhq.lab.eng.bos.redhat.com" --puppet-ca-proxy "rhsm-qe-1.rhq.lab.eng.bos.redhat.com" --environment "production" --puppet-classes stdlib; done;
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Second test was executed in parallel.
[root@rhsm-qe-1 ~]# test_puppet() {
hammer -u admin -p changeme host update --name "test-puppet$1" --puppet-proxy "rhsm-qe-1.rhq.lab.eng.bos.redhat.com" --puppet-ca-proxy "rhsm-qe-1.rhq.lab.eng.bos.redhat.com" --environment "production" --puppet-classes access_insights_client;
}
[root@rhsm-qe-1 ~]# export -f test_puppet
[root@rhsm-qe-1 ~]# parallel test_puppet ::: {1..50}
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
Host updated
parallel can be obtained here: https://sat-r220-02.lab.eng.rdu2.redhat.com/pulp/repos/Sat6-CI/QA/Satellite_RHEL7/content/dist/rhel/server/7/7Server/x86_64/os/repodata/productid.gz:
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:1885 |
Description of problem: 1) Installed 6.2. 2) Regisered 50 content hosts concurently to satellite/capsule 3) Configure puppet on satellite server concurrently for 50 content hosts. hammer -u admin -p changeme host update --name "{{ ansible_fqdn }}" --puppet-proxy "{{ server }}" --puppet-ca-proxy "{{ server }}" --environment "{{ puppet_env }}" --puppet-classes qaredhattest Couple of them failed due to Postgresql ShareLock problems failed: [172.21.0.60 -> satserver.redhat.com] => {"changed": true, "cmd": ["hammer", "-u", "admin", "-p", "changeme", "host", "update", "--name", "docker11capsule60.example.com", "--puppet-proxy", "vmcapsule01.perf.lab.eng.bos.redhat.com", "--puppet-ca-proxy", "vmcapsule01.perf.lab.eng.bos.redhat.com", "--environment", "KT_Default_Organization_Library_test_2", "--puppet-classes", "qaredhattest"], "delta": "0:00:03.004094", "end": "2016-05-13 13:29:37.233191", "rc": 70, "start": "2016-05-13 13:29:34.229097", "warnings": []} stderr: Could not update the host: ERROR: deadlock detected DETAIL: Process 3594 waits for ShareLock on transaction 172545; blocked by process 8097. Process 8097 waits for ExclusiveLock on tuple (0,19) of relation 18248 of database 17944; blocked by process 3594. HINT: See server log for query details. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. concurrently configure puppet for 50 hosts from satellite 6.2 2. 3. Actual results: Some are failing Expected results: All should pass Additional info: