Bug 1374723

Summary: [Improvement] prevent running pulp-manage-db when pulp workers are up
Product: Red Hat Satellite Reporter: Pavel Moravec <pmoravec>
Component: PulpAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED ERRATA QA Contact: Bruno Rocha <rochacbruno>
Severity: high Docs Contact:
Priority: high    
Version: 6.2.0CC: bbuckingham, bkearney, bmbouter, chrobert, daviddavis, dkliban, ggainey, ipanova, jcallaha, ktordeur, mhrivnak, mmccune, omaciel, pcreech, rchan, rochacbruno, ttereshc, xdmoon, zhunting
Target Milestone: UnspecifiedKeywords: Improvement, PrioBumpGSS, Triaged
Target Release: Unused   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: pulp-2.8.7.6-1 Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of:
: 1417079 (view as bug list) Environment:
Last Closed: 2017-03-06 08:28:20 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1385841, 1417079    

Description Pavel Moravec 2016-09-09 13:21:02 UTC
Description of problem:
If pulp-manage-db is executed (due to whatever user/doc error) when pulp workers are running, various collections in mongo can violate their unique indices. Removing the duplicates is quite challenging, esp. because there is no relation model of pulp_database so one does not simply know where all could references to the duplicates be hidden.

This situation is not rare, it happened to 3 customers at least.

Please add to pulp-manage-db a test checking if some worker is running, and if so, stop the tool (or require human confirmation to continue).

See https://pulp.plan.io/issues/2186 raised in upstream.


Version-Release number of selected component (if applicable):
pulp-server-2.8.3.4-1.el7sat.noarch


How reproducible:
100%


Steps to Reproduce:
1. for i in pulp_resource_manager pulp_workers pulp_celerybeat; do service $i start; done
2. pulp-manage-db


Actual results:
2. is executed without a warning or so


Expected results:
2. should be stopped with an error that pulp workers are running, or the tool should require user confirmation to proceed.


Additional info:

Comment 2 pulp-infra@redhat.com 2016-10-05 15:01:31 UTC
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.

Comment 3 pulp-infra@redhat.com 2016-10-05 15:01:35 UTC
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

Comment 4 pulp-infra@redhat.com 2016-10-07 19:31:13 UTC
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.

Comment 5 pulp-infra@redhat.com 2016-10-11 17:01:20 UTC
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

Comment 6 pulp-infra@redhat.com 2016-10-26 05:01:41 UTC
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.

Comment 8 Brian Bouterse 2016-11-21 18:20:34 UTC
*** Bug 1371677 has been marked as a duplicate of this bug. ***

Comment 9 pulp-infra@redhat.com 2016-11-21 21:00:58 UTC
All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.

Comment 10 pulp-infra@redhat.com 2016-12-07 21:01:27 UTC
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.

Comment 11 pulp-infra@redhat.com 2016-12-07 21:31:21 UTC
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 12 pulp-infra@redhat.com 2016-12-19 14:31:50 UTC
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.

Comment 13 pulp-infra@redhat.com 2016-12-30 22:31:53 UTC
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.

Comment 14 pulp-infra@redhat.com 2017-01-04 22:02:10 UTC
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

Comment 15 pulp-infra@redhat.com 2017-01-04 22:32:01 UTC
All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.

Comment 16 pulp-infra@redhat.com 2017-01-18 00:02:21 UTC
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.

Comment 17 pulp-infra@redhat.com 2017-02-01 21:01:54 UTC
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Comment 18 Bruno Rocha 2017-02-15 20:48:08 UTC
Verified in:

satellite-6.2.8-1.0.el7sat.noarch

Results:

[root ~]# for i in pulp_resource_manager pulp_workers pulp_celerybeat; do service $i start; done
Redirecting to /bin/systemctl start  pulp_resource_manager.service
Redirecting to /bin/systemctl start  pulp_workers.service
Redirecting to /bin/systemctl start  pulp_celerybeat.service

[root ~]# sudo -u apache pulp-manage-db
Attempting to connect to localhost:27017
Attempting to connect to localhost:27017
Write concern for Mongo connection: {}

The following processes might still be running:
	scheduler
	resource_manager
	reserved_resource_worker-0
	reserved_resource_worker-1
	reserved_resource_worker-3
	reserved_resource_worker-2
Please wait 1 seconds while Pulp confirms this.  

The following processes are still running, please stop the running workers before retrying the pulp-manage-db command.
 	scheduler
	resource_manager
	reserved_resource_worker-0
	reserved_resource_worker-1
	reserved_resource_worker-3
	reserved_resource_worker-2

Comment 20 errata-xmlrpc 2017-03-06 08:28:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0447