Bug 1655243

Summary: Syncing puppet repo gives Error: PLP0034: The distributor indicated a failed response when publishing repository
Product: Red Hat Satellite Reporter: Nikhil Kathole <nkathole>
Component: Content ManagementAssignee: Justin Sherrill <jsherril>
Status: CLOSED ERRATA QA Contact: Lai <ltran>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.5.0CC: daviddavis, dkliban, jhutar, jsherril, nkathole, oshtaier, pondrejk, zhunting
Target Milestone: 6.5.0Keywords: AutomationBlocker, PrioBumpQA, Regression, Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: tfm-rubygem-katello-3.10.0.9-1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-05-14 12:39:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Nikhil Kathole 2018-12-01 20:18:42 UTC 
Description of problem:
Syncing puppet repo gives 

# hammer -v -u admin -p changeme --output=csv repository synchronize --id="2"
[.........................................................................................................................................] [100%]
Total module count: 12.
Error: PLP0034: The distributor 553e95ec-4cdb-4a39-89e5-2d78c76caab8 indicated a failed response when publishing repository 553e95ec-4cdb-4a39-89e5-2d78c76caab8.


Version-Release number of selected component (if applicable):

Satellite 6.5 snap 6

How reproducible:
always

Steps to Reproduce:
1. # hammer -v -u admin -p changeme --output=csv product create --description="YBEqBOOJibmzeFARmBdQ" --label="VPWRhzUSoLEXpQSROrZx" --name="OfglTcYFuMyFCzYIlCLj" --organization-id="1"
Message,Id,Name
Product created.,270,OfglTcYFuMyFCzYIlCLj

2.# hammer -v -u admin -p changeme --output=csv repository create --content-type="puppet" --name="XMmxqVFlDmAOuyM" --product-id="270" --publish-via-http="true" --url="http://omaciel.fedorapeople.org/bagoftricks"
Message,Id,Name
Repository created.,2,XMmxqVFlDmAOuyM

3.# hammer -v -u admin -p changeme --output=csv repository synchronize --id="2"
[.........................................................................................................................................] [100%]
Total module count: 12.
Error: PLP0034: The distributor 553e95ec-4cdb-4a39-89e5-2d78c76caab8 indicated a failed response when publishing repository 553e95ec-4cdb-4a39-89e5-2d78c76caab8.


Actual results: Sync failed with logs

2018-12-01T15:09:51 [I|kat|] GET: https://satellite/pulp/api/v2/tasks/45db43cf-5347-4c60-a472-bff966589c75/: {"content_type"=>"application/json", "accept"=>"application/json"}
Response: 200: {"exception": null, "task_type": "pulp.server.managers.repo.publish.publish", "_href": "/pulp/api/v2/tasks/45db43cf-5347-4c60-a472-bff966589c75/", "task_id": "45db43cf-5347-4c60-a472-bff966589c75", "tags": ["pulp:repository:553e95ec-4cdb-4a39-89e5-2d78c76caab8", "pulp:action:publish"], "finish_time": "2018-12-01T20:09:50Z", "_ns": "task_status", "start_time": "2018-12-01T20:09:49Z", "traceback": "Traceback (most recent call last):\n  File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 367, in trace_task\n    R = retval = fun(*args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 529, in __call__\n    return super(Task, self).__call__(*args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 107, in __call__\n    return super(PulpTask, self).__call__(*args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 622, in __protected_call__\n    return self.run(*args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py\", line 1110, in publish\n    result = check_publish(repo_obj, dist_id, dist_inst, transfer_repo, conduit, call_config)\n  File \"/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py\", line 1207, in check_publish\n    result = _do_publish(repo_obj, dist_id, dist_inst, transfer_repo, conduit, call_config)\n  File \"/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py\", line 1265, in _do_publish\n    distributor_id=dist_id, summary=publish_report.summary\nPulpCodedException: The distributor 553e95ec-4cdb-4a39-89e5-2d78c76caab8 indicated a failed response when publishing repository 553e95ec-4cdb-4a39-89e5-2d78c76caab8.\n", "spawned_tasks": [], "progress_report": {}, "queue": "reserved_resource_worker-2", "state": "error", "worker_name": "reserved_resource_worker-2@satellite", "result": null, "error": {"code": "PLP0034", "data": {"distributor_id": "553e95ec-4cdb-4a39-89e5-2d78c76caab8", "repo_id": "553e95ec-4cdb-4a39-89e5-2d78c76caab8", "summary": "duplicate unit names"}, "description": "The distributor 553e95ec-4cdb-4a39-89e5-2d78c76caab8 indicated a failed response when publishing repository 553e95ec-4cdb-4a39-89e5-2d78c76caab8.", "sub_errors": []}, "_id": {"$oid": "5c02ea8d2e8c2f8dc1b2e1d7"}, "id": "5c02ea8d2e8c2f8dc1b2e1d7"}
2018-12-01T15:09:51 [E|bac|] PLP0034: The distributor 553e95ec-4cdb-4a39-89e5-2d78c76caab8 indicated a failed response when publishing repository 553e95ec-4cdb-4a39-89e5-2d78c76caab8. (Katello::Errors::PulpError)
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/app/lib/actions/pulp/abstract_async_task.rb:122:in `block in external_task='
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/app/lib/actions/pulp/abstract_async_task.rb:120:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/app/lib/actions/pulp/abstract_async_task.rb:120:in `external_task='
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/app/lib/actions/pulp/repository/sync.rb:34:in `external_task='
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.1/lib/dynflow/action/polling.rb:98:in `poll_external_task_with_rescue'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.1/lib/dynflow/action/polling.rb:21:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.1/lib/dynflow/action/cancellable.rb:13:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/app/lib/actions/pulp/abstract_async_task.rb:45:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.1/lib/dynflow/action.rb:539:in `block (3 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.1/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.1/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.1/lib/dynflow/middleware.rb:31:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.1/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.1/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.1/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'


Expected results:
Successful puppet repo sync. 

Additional info:
Comment 4 Nikhil Kathole 2018-12-03 13:08:54 UTC 
*** Bug 1655563 has been marked as a duplicate of this bug. ***
Comment 7 David Davis 2018-12-04 18:41:41 UTC 
I tested this against 6.5 and it seemed to work for me:

[root@sat-r220-02 ~]# hammer -v -u admin -p changeme --output=csv product create --description="YBEqBOOJibmzeFARmBdQ" --label="VPWRhzUSoLEXpQSROrZx" --name="OfglTcYFuMyFCzYIlCLj" --organization-id="1"
Message,Id,Name
Product created.,277,OfglTcYFuMyFCzYIlCLj
[root@sat-r220-02 ~]# hammer -v -u admin -p changeme --output=csv repository create --content-type="puppet" --name="XMmxqVFlDmAOuyM" --product-id="277" --publish-via-http="true" --url="http://omaciel.fedorapeople.org/bagoftricks"
Message,Id,Name
Repository created.,5,XMmxqVFlDmAOuyM
[root@sat-r220-02 ~]# hammer -v -u admin -p changeme --output=csv repository synchronize --id="5"
[........................................................................................................................................................................................................] [100%]
Total module count: 12.


If you can reproduce this, there should be an Pulp error recorded in system logs with a stacktrace. Or alternatively you can attach a foreman-debug output
Comment 11 David Davis 2018-12-11 17:57:11 UTC 
@dkliban is investigating this issue. Setting NEEDINFO to @dkliban.
Comment 12 Dennis Kliban 2018-12-17 21:57:33 UTC 
I could not reproduce with Pulp 2.16.4 or 2.17.1 or 2.18.0.
Comment 14 David Davis 2018-12-18 12:28:26 UTC 
Setting needinfo.
Comment 17 Justin Sherrill 2018-12-18 18:05:13 UTC 
Created redmine issue https://projects.theforeman.org/issues/25729 from this bug
Comment 18 Satellite Program 2018-12-18 19:01:58 UTC 
Upstream bug assigned to jsherril
Comment 19 Satellite Program 2018-12-18 19:01:59 UTC 
Upstream bug assigned to jsherril
Comment 21 Lai 2019-01-17 18:43:08 UTC 
Follow the steps above and got successful results:

-bash-4.2# hammer -v -u admin -p changeme --output=csv product create --description="YBEqBOOJibmzeFARmBdQ" --label="VPWRhzUSoLEXpQSROrZx" --name="OfglTcYFuMyFCzYIlCLj" --organization-id="1"
Message,Id,Name
Product created.,250,OfglTcYFuMyFCzYIlCLj

-bash-4.2# hammer -v -u admin -p changeme --output=csv repository create --content-type="puppet" --name="XMmxqVFlDmAOuyM" --product-id="250" --publish-via-http="true" --url="http://omaciel.fedorapeople.org/bagoftricks"
Message,Id,Name
Repository created.,41,XMmxqVFlDmAOuyM

-bash-4.2# hammer -v -u admin -p changeme --output=csv repository synchronize --id="41"
[.......................................................................] [100%]
Total module count: 12.
-bash-4.2# 

Verified on 6.5.0 snap 11.  Marking issue as verified.
Comment 24 errata-xmlrpc 2019-05-14 12:39:19 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:1222