Red Hat Bugzilla – Bug 1336007
Postgresql ShareLock problems while configuring puppet concurrently
Last modified: 2016-09-14 17:00:38 EDT
Description of problem: 1) Installed 6.2. 2) Regisered 50 content hosts concurently to satellite/capsule 3) Configure puppet on satellite server concurrently for 50 content hosts. hammer -u admin -p changeme host update --name "{{ ansible_fqdn }}" --puppet-proxy "{{ server }}" --puppet-ca-proxy "{{ server }}" --environment "{{ puppet_env }}" --puppet-classes qaredhattest Couple of them failed due to Postgresql ShareLock problems failed: [172.21.0.60 -> satserver.redhat.com] => {"changed": true, "cmd": ["hammer", "-u", "admin", "-p", "changeme", "host", "update", "--name", "docker11capsule60.example.com", "--puppet-proxy", "vmcapsule01.perf.lab.eng.bos.redhat.com", "--puppet-ca-proxy", "vmcapsule01.perf.lab.eng.bos.redhat.com", "--environment", "KT_Default_Organization_Library_test_2", "--puppet-classes", "qaredhattest"], "delta": "0:00:03.004094", "end": "2016-05-13 13:29:37.233191", "rc": 70, "start": "2016-05-13 13:29:34.229097", "warnings": []} stderr: Could not update the host: ERROR: deadlock detected DETAIL: Process 3594 waits for ShareLock on transaction 172545; blocked by process 8097. Process 8097 waits for ExclusiveLock on tuple (0,19) of relation 18248 of database 17944; blocked by process 3594. HINT: See server log for query details. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. concurrently configure puppet for 50 hosts from satellite 6.2 2. 3. Actual results: Some are failing Expected results: All should pass Additional info:
Upstream bug component is Infrastructure
Moving to POST since upstream bug http://projects.theforeman.org/issues/14076 has been closed
Verified in Satellite 6.2.2 Snap 1.1 Performed the setup using docker content hosts. First test was executed in serial. [root@rhsm-qe-1 ~]# for i in {1..50}; do hammer -u admin -p changeme host update --name "test-puppet$i" --puppet-proxy "rhsm-qe-1.rhq.lab.eng.bos.redhat.com" --puppet-ca-proxy "rhsm-qe-1.rhq.lab.eng.bos.redhat.com" --environment "production" --puppet-classes stdlib; done; Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Second test was executed in parallel. [root@rhsm-qe-1 ~]# test_puppet() { hammer -u admin -p changeme host update --name "test-puppet$1" --puppet-proxy "rhsm-qe-1.rhq.lab.eng.bos.redhat.com" --puppet-ca-proxy "rhsm-qe-1.rhq.lab.eng.bos.redhat.com" --environment "production" --puppet-classes access_insights_client; } [root@rhsm-qe-1 ~]# export -f test_puppet [root@rhsm-qe-1 ~]# parallel test_puppet ::: {1..50} Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated Host updated parallel can be obtained here: https://sat-r220-02.lab.eng.rdu2.redhat.com/pulp/repos/Sat6-CI/QA/Satellite_RHEL7/content/dist/rhel/server/7/7Server/x86_64/os/repodata/productid.gz:
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:1885