Bug 1800681 - After using the /associate endpoint, publish of the new repository fails
Summary: After using the /associate endpoint, publish of the new repository fails
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Pulp
Version: 6.7.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: 6.7.0
Assignee: satellite6-bugs
QA Contact: Bruno Rocha
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-02-07 17:02 UTC by Partha Aji
Modified: 2020-10-05 14:08 UTC (History)
9 users (show)

Fixed In Version: pulp-rpm-2.21.0.3-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-04-14 13:28:43 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Pulp Redmine 6096 Normal CLOSED - CURRENTRELEASE After using the /associate endpoint, publish of the new repository fails 2020-03-05 18:47:53 UTC
Red Hat Product Errata RHSA-2020:1454 None None None 2020-04-14 13:28:53 UTC

Description Partha Aji 2020-02-07 17:02:22 UTC
If you generate a new repository using the associate command, and then attempt to publish that repository, it fails with the following stacktrace as a result of ModulemdDefaults in the new repo not having a checksum:

Jan 31 21:12:20 pulp2.dev pulp[11905]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[8e1ad085-9190-48d8-8b3b-4dd18d00813f]
Jan 31 21:12:20 pulp2.dev pulp[13062]: pulp.server.controllers.repository:ERROR: [8d22564d] (13062-22624) Exception caught from plugin during publish for repo [destination5]
Jan 31 21:12:20 pulp2.dev pulp[13062]: pulp.server.controllers.repository:ERROR: [8d22564d] (13062-22624) Traceback (most recent call last):
Jan 31 21:12:20 pulp2.dev pulp[13062]: pulp.server.controllers.repository:ERROR: [8d22564d] (13062-22624)   File "/home/vagrant/devel/pulp/server/pulp/server/controllers/repository.py", line 1259, in _do_publish
Jan 31 21:12:20 pulp2.dev pulp[13062]: pulp.server.controllers.repository:ERROR: [8d22564d] (13062-22624)     publish_report = publish_repo(transfer_repo, conduit, call_config)
Jan 31 21:12:20 pulp2.dev pulp[13062]: pulp.server.controllers.repository:ERROR: [8d22564d] (13062-22624)   File "/home/vagrant/devel/pulp/server/pulp/server/async/tasks.py", line 894, in wrap_f
Jan 31 21:12:20 pulp2.dev pulp[13062]: pulp.server.controllers.repository:ERROR: [8d22564d] (13062-22624)     return f(*args, **kwargs)
Jan 31 21:12:20 pulp2.dev pulp[13062]: pulp.server.controllers.repository:ERROR: [8d22564d] (13062-22624)   File "/home/vagrant/devel/pulp_rpm/plugins/pulp_rpm/plugins/distributors/yum/distributor.py", line 174, in publish_repo
Jan 31 21:12:20 pulp2.dev pulp[13062]: pulp.server.controllers.repository:ERROR: [8d22564d] (13062-22624)     return self._publisher.process_lifecycle()
Jan 31 21:12:20 pulp2.dev pulp[13062]: pulp.server.controllers.repository:ERROR: [8d22564d] (13062-22624)   File "/home/vagrant/devel/pulp/server/pulp/plugins/util/publish_step.py", line 573, in process_lifecycle
Jan 31 21:12:20 pulp2.dev pulp[13062]: pulp.server.controllers.repository:ERROR: [8d22564d] (13062-22624)     super(PluginStep, self).process_lifecycle()
Jan 31 21:12:20 pulp2.dev pulp[13062]: pulp.server.controllers.repository:ERROR: [8d22564d] (13062-22624)   File "/home/vagrant/devel/pulp/server/pulp/plugins/util/publish_step.py", line 164, in process_lifecycle
Jan 31 21:12:20 pulp2.dev pulp[13062]: pulp.server.controllers.repository:ERROR: [8d22564d] (13062-22624)     step.process()
Jan 31 21:12:20 pulp2.dev pulp[13062]: pulp.server.controllers.repository:ERROR: [8d22564d] (13062-22624)   File "/home/vagrant/devel/pulp/server/pulp/plugins/util/publish_step.py", line 240, in process
Jan 31 21:12:20 pulp2.dev pulp[13062]: pulp.server.controllers.repository:ERROR: [8d22564d] (13062-22624)     self._process_block(item=item)
Jan 31 21:12:20 pulp2.dev pulp[13062]: pulp.server.controllers.repository:ERROR: [8d22564d] (13062-22624)   File "/home/vagrant/devel/pulp/server/pulp/plugins/util/publish_step.py", line 302, in _process_block
Jan 31 21:12:20 pulp2.dev pulp[13062]: pulp.server.controllers.repository:ERROR: [8d22564d] (13062-22624)     self.process_main(item=item)
Jan 31 21:12:20 pulp2.dev pulp[13062]: pulp.server.controllers.repository:ERROR: [8d22564d] (13062-22624)   File "/home/vagrant/devel/pulp_rpm/plugins/pulp_rpm/plugins/distributors/yum/publish.py", line 896, in process_main
Jan 31 21:12:20 pulp2.dev pulp[13062]: pulp.server.controllers.repository:ERROR: [8d22564d] (13062-22624)    self.context.add_document(document, doc_checksum=item.checksum)
Jan 31 21:12:20 pulp2.dev pulp[13062]: pulp.server.controllers.repository:ERROR: [8d22564d] (13062-22624)   File "/home/vagrant/devel/pulp_rpm/plugins/pulp_rpm/plugins/distributors/yum/metadata/modules.py", line 37, in add_document
Jan 31 21:12:20 pulp2.dev pulp[13062]: pulp.server.controllers.repository:ERROR: [8d22564d] (13062-22624)     raise PulpCodedException(error_codes.RPM1017)
Jan 31 21:12:20 pulp2.dev pulp[13062]: pulp.server.controllers.repository:ERROR: [8d22564d] (13062-22624) PulpCodedException: Publish modules.yaml failed: modulemd checksum mismatch detected.
Jan 31 21:12:20 pulp2.dev pulp[13062]: pulp.server.async.tasks:INFO: [8d22564d] Task failed : [8d22564d-842a-48d5-870a-db4b166c5614] : Publish modules.yaml failed: modulemd checksum mismatch detected.



You can reproduce this behavior with the following steps:
"""
export BASE1='foo'
export BASE2='bar'
export REMOTE1='https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-with-modules-modified/'
export REMOTE2='https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-unsigned-modified/'
export DEST='destination'
export ERRATUM='RHEA-2012:0059'
pulp-admin rpm repo create --repo-id=$BASE1 --relative-url=$BASE1 --feed=$REMOTE1
pulp-admin rpm repo sync run --repo-id=$BASE1
pulp-admin rpm repo create --repo-id=$BASE2 --relative-url=$BASE2 --feed=$REMOTE2
pulp-admin rpm repo sync run --repo-id=$BASE2
pulp-admin rpm repo create --repo-id=$DEST
TASK_OUTPUT=$(curl -k -u admin:admin --cert ~/.pulp/user-cert.pem \
         -d "{\"source_repo_id\":\"$BASE1\",\"criteria\":{\"type_ids\":[\"erratum\"],\"filters\":{\"unit\":{\"id\":\"$ERRATUM\"}}},\"override_config\":{\"recursive_conservative\":true,\"additional_repos\":{\"$BASE2\": \"$DEST\"}}}" \
         -H "Content-Type: application/json" \
         -X POST https://localhost/pulp/api/v2/repositories/$DEST/actions/associate/)
TASK_URL=$(echo $TASK_OUTPUT | jq -r '.spawned_tasks[]._href')
echo "Task url : " $TASK_URL
if [ -z "$TASK_URL" ]; then echo "NO TASK?!?"; exit; fi
wait_until_task_finished https://localhost$TASK_URL
pulp-admin rpm repo publish run --repo-id $DEST
"""

Comment 3 Partha Aji 2020-02-07 17:06:00 UTC
From a sat perpective -
This is  a non issue for regular publishes/publsih with depsolve. Its more a case of investigating if incremental update will hit this and fix. It is possible that cdn adds a new module with module default and if we say incremental update this errata that points to the new stuff  we may hit into issue.                                                                                                                                  
Will be good to have this fix in any case from pulp.

Comment 4 pulp-infra@redhat.com 2020-02-07 17:31:55 UTC
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.

Comment 5 pulp-infra@redhat.com 2020-02-07 17:31:57 UTC
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

Comment 6 pulp-infra@redhat.com 2020-02-11 15:43:09 UTC
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

Comment 7 pulp-infra@redhat.com 2020-02-11 16:02:26 UTC
All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.

Comment 9 pulp-infra@redhat.com 2020-02-27 17:02:31 UTC
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.

Comment 10 pulp-infra@redhat.com 2020-03-05 18:47:54 UTC
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Comment 12 Lai 2020-03-30 21:42:51 UTC
Step to test:

1. ssh into sat
2. install pulp-admin
3. run script:

export BASE1='foo'
export BASE2='bar'
export REMOTE1='https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-with-modules-modified/'
export REMOTE2='https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-unsigned-modified/'
export DEST1='destination1'
export DEST2='destination2'
export ERRATUM='RHEA-2012:0059'
pulp-admin rpm repo create --repo-id=$BASE1 --relative-url=$BASE1 --feed=$REMOTE1
pulp-admin rpm repo sync run --repo-id=$BASE1
pulp-admin rpm repo create --repo-id=$BASE2 --relative-url=$BASE2 --feed=$REMOTE2
pulp-admin rpm repo sync run --repo-id=$BASE2
pulp-admin rpm repo create --repo-id=$DEST1
pulp-admin rpm repo create --repo-id=$DEST2
TASK_OUTPUT=$(curl -k -u admin:admin --cert ~/.pulp/user-cert.pem \
         -d "{\"source_repo_id\":\"$BASE1\",\"criteria\":{\"type_ids\":[\"erratum\"],\"filters\":{\"unit\":{\"id\":\"$ERRATUM\"}}},\"override_config\":{\"recursive_conservative\":true,\"additional_repos\":{\"$BASE2\": \"$DEST2\"}}}" \
         -H "Content-Type: application/json" \
         -X POST https://localhost/pulp/api/v2/repositories/$DEST1/actions/associate/)
TASK_URL=$(echo $TASK_OUTPUT | jq -r '.spawned_tasks[]._href')
echo "Task url : " $TASK_URL
if [ -z "$TASK_URL" ]; then echo "NO TASK?!?"; exit; fi
wait_until_task_finished https://localhost$TASK_URL
pulp-admin rpm repo publish run --repo-id $DEST1
pulp-admin rpm repo publish run --repo-id $DEST2

4. check the log via /var/log/messages and see that there's no errors.
5. cat out  /var/lib/pulp/published/yum/https/repos/destination1/repodata/*modules.yaml*

4)
Expected:
Mar 30 17:35:41 dhcp-2-51 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[3daddb3e-5ee9-4dfd-b6eb-259f162ca01b]
Mar 30 17:35:41 dhcp-2-51 pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.publish.publish[bdd7e9e1-e7b7-4630-bb06-f75608315c02]
Mar 30 17:35:41 dhcp-2-51 pulp: celery.app.trace:INFO: [3daddb3e] Task pulp.server.async.tasks._queue_reserved_task[3daddb3e-5ee9-4dfd-b6eb-259f162ca01b] succeeded in 0.0318934479728s: None
Mar 30 17:35:41 dhcp-2-51 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[c0029bd4-b539-48d2-afc8-083216ea45f9]
Mar 30 17:35:41 dhcp-2-51 pulp: celery.app.trace:INFO: [bdd7e9e1] Task pulp.server.managers.repo.publish.publish[bdd7e9e1-e7b7-4630-bb06-f75608315c02] succeeded in 0.233778246038s: {'exception': None, 'repo_id': 'destination1', 'traceback': None, 'started': '2020-03-30T21:35:41Z', '_ns': 'repo_publish_results', 'completed': datetime.datetime(2020, 3, 30, 21, 35, 41, 898107, tzinfo=<isodate.tzinfo.Utc object at 0x7f03b2f7ac50>), 'error_message': None, 'distributor_type_id': 'yum_distributor', 'distributor_id': 'yum_distributor', 'summary': {'generate sqlite': 'SKIPPED', 'initialize_repo_metadata': 'FINISHED', 'remove_old_repodata': 'FINISHED', 'rpms': 'FINISHED', 'modules': 'FINISHED', 'close_repo_metadata': 'FINISHED', 'drpms': 'SKIPPED', 'comps': 'FINISHED', 'distribution': 'FINISHED', 'repoview': 'SKIPPED', 'publish_directory': 'FINISHED', 'errata': 'FINISHED', 'metadata': 'FINISHED'}, 'result': 'success', 'id': '5e82662d067da34ed6b30328', 'details': [{'num_processed': 1, 'items_total': 1, 'state': 'FINISHED', 'num_success': 1, 'error_details': [...], 'description': 'Initializing repo metadata', 'num_failures': 0, 'step_id': '76ddf5ea-6e35-440e-af7e-5932d8dd9ca9', 'step_type': 'initial...', ...}]}
Mar 30 17:35:41 dhcp-2-51 pulp: celery.app.trace:INFO: [c0029bd4] Task pulp.server.async.tasks._release_resource[c0029bd4-b539-48d2-afc8-083216ea45f9] succeeded in 0.00185615505325s: None
Mar 30 17:35:44 dhcp-2-51 dhclient[1664]: DHCPREQUEST on eth0 to 10.11.5.19 port 67 (xid=0x171c8c16)
Mar 30 17:35:46 dhcp-2-51 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[aa55d767-84cb-4f79-bd31-f5967a151a8b]
Mar 30 17:35:46 dhcp-2-51 pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.publish.publish[3e0d777f-f071-43b1-9b1f-b5c1a42862fd]
Mar 30 17:35:46 dhcp-2-51 pulp: celery.app.trace:INFO: [aa55d767] Task pulp.server.async.tasks._queue_reserved_task[aa55d767-84cb-4f79-bd31-f5967a151a8b] succeeded in 0.0274146299926s: None
Mar 30 17:35:46 dhcp-2-51 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[cc7a8de5-d663-4a4e-8f0f-34d35d969600]
Mar 30 17:35:46 dhcp-2-51 pulp: celery.app.trace:INFO: [3e0d777f] Task pulp.server.managers.repo.publish.publish[3e0d777f-f071-43b1-9b1f-b5c1a42862fd] succeeded in 0.163032072014s: {'exception': None, 'repo_id': 'destination2', 'traceback': None, 'started': '2020-03-30T21:35:46Z', '_ns': 'repo_publish_results', 'completed': datetime.datetime(2020, 3, 30, 21, 35, 46, 735665, tzinfo=<isodate.tzinfo.Utc object at 0x7f03b2f7ac50>), 'error_message': None, 'distributor_type_id': 'yum_distributor', 'distributor_id': 'yum_distributor', 'summary': {'generate sqlite': 'SKIPPED', 'initialize_repo_metadata': 'FINISHED', 'remove_old_repodata': 'FINISHED', 'rpms': 'FINISHED', 'modules': 'SKIPPED', 'close_repo_metadata': 'FINISHED', 'drpms': 'SKIPPED', 'comps': 'FINISHED', 'distribution': 'FINISHED', 'repoview': 'SKIPPED', 'publish_directory': 'FINISHED', 'errata': 'FINISHED', 'metadata': 'FINISHED'}, 'result': 'success', 'id': '5e826632067da34ed6b30329', 'details': [{'num_processed': 1, 'items_total': 1, 'state': 'FINISHED', 'num_success': 1, 'error_details': [...], 'description': 'Initializing repo metadata', 'num_failures': 0, 'step_id': 'e9192bac-823f-402e-82f3-3ce744605501', 'step_type': 'initiali...', ...}]}
Mar 30 17:35:46 dhcp-2-51 pulp: celery.app.trace:INFO: [cc7a8de5] Task pulp.server.async.tasks._release_resource[cc7a8de5-d663-4a4e-8f0f-34d35d969600] succeeded in 0.00176209799247s: None
Mar 30 17:36:03 dhcp-2-51 dhclient[1664]: DHCPREQUEST on eth0 to 10.11.5.19 port 67 (xid=0x171c8c16)

Actual:
Mar 30 17:35:41 dhcp-2-51 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[3daddb3e-5ee9-4dfd-b6eb-259f162ca01b]
Mar 30 17:35:41 dhcp-2-51 pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.publish.publish[bdd7e9e1-e7b7-4630-bb06-f75608315c02]
Mar 30 17:35:41 dhcp-2-51 pulp: celery.app.trace:INFO: [3daddb3e] Task pulp.server.async.tasks._queue_reserved_task[3daddb3e-5ee9-4dfd-b6eb-259f162ca01b] succeeded in 0.0318934479728s: None
Mar 30 17:35:41 dhcp-2-51 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[c0029bd4-b539-48d2-afc8-083216ea45f9]
Mar 30 17:35:41 dhcp-2-51 pulp: celery.app.trace:INFO: [bdd7e9e1] Task pulp.server.managers.repo.publish.publish[bdd7e9e1-e7b7-4630-bb06-f75608315c02] succeeded in 0.233778246038s: {'exception': None, 'repo_id': 'destination1', 'traceback': None, 'started': '2020-03-30T21:35:41Z', '_ns': 'repo_publish_results', 'completed': datetime.datetime(2020, 3, 30, 21, 35, 41, 898107, tzinfo=<isodate.tzinfo.Utc object at 0x7f03b2f7ac50>), 'error_message': None, 'distributor_type_id': 'yum_distributor', 'distributor_id': 'yum_distributor', 'summary': {'generate sqlite': 'SKIPPED', 'initialize_repo_metadata': 'FINISHED', 'remove_old_repodata': 'FINISHED', 'rpms': 'FINISHED', 'modules': 'FINISHED', 'close_repo_metadata': 'FINISHED', 'drpms': 'SKIPPED', 'comps': 'FINISHED', 'distribution': 'FINISHED', 'repoview': 'SKIPPED', 'publish_directory': 'FINISHED', 'errata': 'FINISHED', 'metadata': 'FINISHED'}, 'result': 'success', 'id': '5e82662d067da34ed6b30328', 'details': [{'num_processed': 1, 'items_total': 1, 'state': 'FINISHED', 'num_success': 1, 'error_details': [...], 'description': 'Initializing repo metadata', 'num_failures': 0, 'step_id': '76ddf5ea-6e35-440e-af7e-5932d8dd9ca9', 'step_type': 'initial...', ...}]}
Mar 30 17:35:41 dhcp-2-51 pulp: celery.app.trace:INFO: [c0029bd4] Task pulp.server.async.tasks._release_resource[c0029bd4-b539-48d2-afc8-083216ea45f9] succeeded in 0.00185615505325s: None
Mar 30 17:35:44 dhcp-2-51 dhclient[1664]: DHCPREQUEST on eth0 to 10.11.5.19 port 67 (xid=0x171c8c16)
Mar 30 17:35:46 dhcp-2-51 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[aa55d767-84cb-4f79-bd31-f5967a151a8b]
Mar 30 17:35:46 dhcp-2-51 pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.publish.publish[3e0d777f-f071-43b1-9b1f-b5c1a42862fd]
Mar 30 17:35:46 dhcp-2-51 pulp: celery.app.trace:INFO: [aa55d767] Task pulp.server.async.tasks._queue_reserved_task[aa55d767-84cb-4f79-bd31-f5967a151a8b] succeeded in 0.0274146299926s: None
Mar 30 17:35:46 dhcp-2-51 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[cc7a8de5-d663-4a4e-8f0f-34d35d969600]
Mar 30 17:35:46 dhcp-2-51 pulp: celery.app.trace:INFO: [3e0d777f] Task pulp.server.managers.repo.publish.publish[3e0d777f-f071-43b1-9b1f-b5c1a42862fd] succeeded in 0.163032072014s: {'exception': None, 'repo_id': 'destination2', 'traceback': None, 'started': '2020-03-30T21:35:46Z', '_ns': 'repo_publish_results', 'completed': datetime.datetime(2020, 3, 30, 21, 35, 46, 735665, tzinfo=<isodate.tzinfo.Utc object at 0x7f03b2f7ac50>), 'error_message': None, 'distributor_type_id': 'yum_distributor', 'distributor_id': 'yum_distributor', 'summary': {'generate sqlite': 'SKIPPED', 'initialize_repo_metadata': 'FINISHED', 'remove_old_repodata': 'FINISHED', 'rpms': 'FINISHED', 'modules': 'SKIPPED', 'close_repo_metadata': 'FINISHED', 'drpms': 'SKIPPED', 'comps': 'FINISHED', 'distribution': 'FINISHED', 'repoview': 'SKIPPED', 'publish_directory': 'FINISHED', 'errata': 'FINISHED', 'metadata': 'FINISHED'}, 'result': 'success', 'id': '5e826632067da34ed6b30329', 'details': [{'num_processed': 1, 'items_total': 1, 'state': 'FINISHED', 'num_success': 1, 'error_details': [...], 'description': 'Initializing repo metadata', 'num_failures': 0, 'step_id': 'e9192bac-823f-402e-82f3-3ce744605501', 'step_type': 'initiali...', ...}]}
Mar 30 17:35:46 dhcp-2-51 pulp: celery.app.trace:INFO: [cc7a8de5] Task pulp.server.async.tasks._release_resource[cc7a8de5-d663-4a4e-8f0f-34d35d969600] succeeded in 0.00176209799247s: None
Mar 30 17:36:03 dhcp-2-51 dhclient[1664]: DHCPREQUEST on eth0 to 10.11.5.19 port 67 (xid=0x171c8c16)


5)
Expected:
File should only contains duck and kangaroo

Actual:
File only contains duck and kangaroo

Marking issue as verified on 6.7.0_17

Comment 14 errata-xmlrpc 2020-04-14 13:28:43 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:1454


Note You need to log in before you can comment on or make changes to this bug.