Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1473595 - Messages in resource manager queue are not persisted after restarting Qpid and cause tasks never start
Summary: Messages in resource manager queue are not persisted after restarting Qpid an...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Pulp
Version: 6.2.10
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: Jitendra Yejare
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-07-21 09:00 UTC by Hao Chang Yu
Modified: 2022-03-13 14:21 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-02-21 16:54:37 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Pulp Redmine 2927 0 High CLOSED - CURRENTRELEASE Messages in resource manager queue are not persisted after restarting Qpid and cause tasks never start 2017-10-17 14:02:08 UTC

Description Hao Chang Yu 2017-07-21 09:00:10 UTC 
Description of problem:
The messages in resource manager will be lost after restarting Qpid broker and cause tasks in "waiting" state forever.

Steps to Reproduce:
1. systemctl stop pulp_resource_manager
2. Sync a repo in the Satellite web ui
3. systemctl restart qpidd
4. systemctl start pulp_resource_manager

Before restarting Qpid:
queue                                            dur  autoDel  excl  msg   msgIn  msgOut  bytes  bytesIn  bytesOut  cons  bind
================================================================================================================================
 resource_manager                                 Y                      1    20     19    1.59k  32.6k    31.0k        0     2

msg is 1

After restarting Qpid:
queue                                            dur  autoDel  excl  msg   msgIn  msgOut  bytes  bytesIn  bytesOut  cons  bind
================================================================================================================================
resource_manager                                 Y                      0     0      0       0      0        0         0     2

msg is now 0


Actual results:
Tasks in waiting state forever

Expected results:
Tasks should proceed after restarting Qpid.
Comment 1 David Davis 2017-07-21 15:13:05 UTC 
The issue in https://pulp.plan.io/issues/2861 is caused by resource_manager removing its task from the queue during a warm shutdown. 

This issue is different in that resource manager is already shutdown. I have opened a new upstream issue:

https://pulp.plan.io/issues/2927
Comment 2 pulp-infra@redhat.com 2017-07-21 15:31:52 UTC 
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.
Comment 3 pulp-infra@redhat.com 2017-07-21 15:31:55 UTC 
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.
Comment 5 pulp-infra@redhat.com 2017-08-04 15:01:46 UTC 
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.
Comment 6 pulp-infra@redhat.com 2017-08-18 03:20:56 UTC 
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.
Comment 7 pulp-infra@redhat.com 2017-09-11 19:46:30 UTC 
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.
Comment 8 pulp-infra@redhat.com 2017-09-11 20:07:45 UTC 
All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.
Comment 9 pulp-infra@redhat.com 2017-10-17 14:02:09 UTC 
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.
Comment 10 Jitendra Yejare 2017-12-20 11:12:07 UTC 
Verifying this bug as per steps mentioned in the description and some observations are :

After stop pulp_resource_manager I am trying sync repo in satellite but somehow the sync is errored with the message below and was no more in pending state:
```
There was an issue with the backend service pulp: Not all necessary pulp workers running at https://qeblade36.rhq.lab.eng.bos.redhat.com/pulp/api/v2/.
```
hence the sync doesn't start. So is this expected?

Still, I restarted qpidd and check the queue before and after the restart.
After restart, I see its 0.

On starting pulp_resource_manager, the repo sync did not resume as the task was in error state. So Is this expected? I doubt.

So does all this behavior is correct ?
May I mark this bug verified?
Comment 11 David Davis 2017-12-20 15:24:32 UTC 
Katello has checks in place so that it doesn't send tasks to pulp unless all pulp processes are running. So the steps in the original comment aren't going to work. To bypass the checks, we can just use pulp-admin to do the sync.

First install and setup pulp-admin per this:

https://access.redhat.com/solutions/1295653


Now run these steps to verify this bug:

systemctl stop pulp_resource_manager

pulpAdminPassword=$(grep ^default_password /etc/pulp/server.conf | cut -d' ' -f2)

pulp-admin -u admin -p $pulpAdminPassword rpm repo sync run --repo-id 58ec9596-497b-42ea-acf3-6580062e0924

systemctl restart qpidd

systemctl start pulp_resource_manager



To verify this bug, run this and look at the last sync task:

pulp-admin -u admin -p $pulpAdminPassword tasks list -a


You should see something like this with either the state Running or Successful:

Operations:  sync
Resources:   58ec9596-497b-42ea-acf3-6580062e0924 (repository)
State:       Running
Start Time:  2017-12-20T15:22:53Z
Finish Time: Incomplete
Task Id:     b97c7b82-141a-4c76-8abb-7146d1f1d58a
Comment 12 Jitendra Yejare 2017-12-20 16:03:00 UTC 
Verified !

@ Satellite 6.3 snap 29

As per comment 11, installed pulp-admin in satellite to test this bug else its difficult to test this bug.

Steps:
(on Satellite)

1. systemctl stop pulp_resource_manager
2. pulpAdminPassword=$(grep ^default_password /etc/pulp/server.conf | cut -d' ' -f2)
3. pulp-admin -u admin -p $pulpAdminPassword rpm repo sync run --repo-id <repo_uuid>
```
+----------------------------------------------------------------------+
    Synchronizing Repository [58ec9596-497b-42ea-acf3-6580062esa887]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.


[/]
Waiting to begin...
```
-> Now the sync task in pending state as pulp manager is stopped
4. Now restart qpidd daemon
5. systemctl start pulp_resource_manager
-> Now the sync task should start running
```
-- Continue from where it stopped earlier
Downloading metadata...
[\]
... completed

Downloading repository content...
[-]
[==================================================] 100%
RPMs:       0/0 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[==================================================] 100%
Distributions: 0/0 items
... completed

Importing errata...
[-]
... completed

Importing package groups/categories...
[-]
... completed

Cleaning duplicate packages...
[-]
... completed

Task Succeeded

Task Succeeded
```



Qpid_Queue Before qpidd restart:
```
resource_manager                                                               Y                      1     3      2    1.32k  4.04k    2.72k        0     2
```

Qpid_Queue After qpidd restart:
```
resource_manager                                                               Y                      1     1      0    1.32k  1.32k       0         0     2
```

Hence, Moving the state to Verified.
Comment 13 Satellite Program 2018-02-21 16:54:37 UTC 
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA.
> > 
> > For information on the advisory, and where to find the updated files, follow the link below.
> > 
> > If the solution does not work for you, open a new bug report.
> > 
> > https://access.redhat.com/errata/RHSA-2018:0336
Note You need to log in before you can comment on or make changes to this bug.