Bug 1800681

Summary: After using the /associate endpoint, publish of the new repository fails
Product: Red Hat Satellite Reporter: Partha Aji <paji>
Component: PulpAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED ERRATA QA Contact: Bruno Rocha <rochacbruno>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 6.7.0CC: bmbouter, daviddavis, dkliban, ggainey, ipanova, ktordeur, ltran, rchan, ttereshc, zhunting
Target Milestone: 6.7.0Keywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: pulp-rpm-2.21.0.3-1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-04-14 13:28:43 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Partha Aji 2020-02-07 17:02:22 UTC
If you generate a new repository using the associate command, and then attempt to publish that repository, it fails with the following stacktrace as a result of ModulemdDefaults in the new repo not having a checksum:

Jan 31 21:12:20 pulp2.dev pulp[11905]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[8e1ad085-9190-48d8-8b3b-4dd18d00813f]
Jan 31 21:12:20 pulp2.dev pulp[13062]: pulp.server.controllers.repository:ERROR: [8d22564d] (13062-22624) Exception caught from plugin during publish for repo [destination5]
Jan 31 21:12:20 pulp2.dev pulp[13062]: pulp.server.controllers.repository:ERROR: [8d22564d] (13062-22624) Traceback (most recent call last):
Jan 31 21:12:20 pulp2.dev pulp[13062]: pulp.server.controllers.repository:ERROR: [8d22564d] (13062-22624)   File "/home/vagrant/devel/pulp/server/pulp/server/controllers/repository.py", line 1259, in _do_publish
Jan 31 21:12:20 pulp2.dev pulp[13062]: pulp.server.controllers.repository:ERROR: [8d22564d] (13062-22624)     publish_report = publish_repo(transfer_repo, conduit, call_config)
Jan 31 21:12:20 pulp2.dev pulp[13062]: pulp.server.controllers.repository:ERROR: [8d22564d] (13062-22624)   File "/home/vagrant/devel/pulp/server/pulp/server/async/tasks.py", line 894, in wrap_f
Jan 31 21:12:20 pulp2.dev pulp[13062]: pulp.server.controllers.repository:ERROR: [8d22564d] (13062-22624)     return f(*args, **kwargs)
Jan 31 21:12:20 pulp2.dev pulp[13062]: pulp.server.controllers.repository:ERROR: [8d22564d] (13062-22624)   File "/home/vagrant/devel/pulp_rpm/plugins/pulp_rpm/plugins/distributors/yum/distributor.py", line 174, in publish_repo
Jan 31 21:12:20 pulp2.dev pulp[13062]: pulp.server.controllers.repository:ERROR: [8d22564d] (13062-22624)     return self._publisher.process_lifecycle()
Jan 31 21:12:20 pulp2.dev pulp[13062]: pulp.server.controllers.repository:ERROR: [8d22564d] (13062-22624)   File "/home/vagrant/devel/pulp/server/pulp/plugins/util/publish_step.py", line 573, in process_lifecycle
Jan 31 21:12:20 pulp2.dev pulp[13062]: pulp.server.controllers.repository:ERROR: [8d22564d] (13062-22624)     super(PluginStep, self).process_lifecycle()
Jan 31 21:12:20 pulp2.dev pulp[13062]: pulp.server.controllers.repository:ERROR: [8d22564d] (13062-22624)   File "/home/vagrant/devel/pulp/server/pulp/plugins/util/publish_step.py", line 164, in process_lifecycle
Jan 31 21:12:20 pulp2.dev pulp[13062]: pulp.server.controllers.repository:ERROR: [8d22564d] (13062-22624)     step.process()
Jan 31 21:12:20 pulp2.dev pulp[13062]: pulp.server.controllers.repository:ERROR: [8d22564d] (13062-22624)   File "/home/vagrant/devel/pulp/server/pulp/plugins/util/publish_step.py", line 240, in process
Jan 31 21:12:20 pulp2.dev pulp[13062]: pulp.server.controllers.repository:ERROR: [8d22564d] (13062-22624)     self._process_block(item=item)
Jan 31 21:12:20 pulp2.dev pulp[13062]: pulp.server.controllers.repository:ERROR: [8d22564d] (13062-22624)   File "/home/vagrant/devel/pulp/server/pulp/plugins/util/publish_step.py", line 302, in _process_block
Jan 31 21:12:20 pulp2.dev pulp[13062]: pulp.server.controllers.repository:ERROR: [8d22564d] (13062-22624)     self.process_main(item=item)
Jan 31 21:12:20 pulp2.dev pulp[13062]: pulp.server.controllers.repository:ERROR: [8d22564d] (13062-22624)   File "/home/vagrant/devel/pulp_rpm/plugins/pulp_rpm/plugins/distributors/yum/publish.py", line 896, in process_main
Jan 31 21:12:20 pulp2.dev pulp[13062]: pulp.server.controllers.repository:ERROR: [8d22564d] (13062-22624)    self.context.add_document(document, doc_checksum=item.checksum)
Jan 31 21:12:20 pulp2.dev pulp[13062]: pulp.server.controllers.repository:ERROR: [8d22564d] (13062-22624)   File "/home/vagrant/devel/pulp_rpm/plugins/pulp_rpm/plugins/distributors/yum/metadata/modules.py", line 37, in add_document
Jan 31 21:12:20 pulp2.dev pulp[13062]: pulp.server.controllers.repository:ERROR: [8d22564d] (13062-22624)     raise PulpCodedException(error_codes.RPM1017)
Jan 31 21:12:20 pulp2.dev pulp[13062]: pulp.server.controllers.repository:ERROR: [8d22564d] (13062-22624) PulpCodedException: Publish modules.yaml failed: modulemd checksum mismatch detected.
Jan 31 21:12:20 pulp2.dev pulp[13062]: pulp.server.async.tasks:INFO: [8d22564d] Task failed : [8d22564d-842a-48d5-870a-db4b166c5614] : Publish modules.yaml failed: modulemd checksum mismatch detected.



You can reproduce this behavior with the following steps:
"""
export BASE1='foo'
export BASE2='bar'
export REMOTE1='https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-with-modules-modified/'
export REMOTE2='https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-unsigned-modified/'
export DEST='destination'
export ERRATUM='RHEA-2012:0059'
pulp-admin rpm repo create --repo-id=$BASE1 --relative-url=$BASE1 --feed=$REMOTE1
pulp-admin rpm repo sync run --repo-id=$BASE1
pulp-admin rpm repo create --repo-id=$BASE2 --relative-url=$BASE2 --feed=$REMOTE2
pulp-admin rpm repo sync run --repo-id=$BASE2
pulp-admin rpm repo create --repo-id=$DEST
TASK_OUTPUT=$(curl -k -u admin:admin --cert ~/.pulp/user-cert.pem \
         -d "{\"source_repo_id\":\"$BASE1\",\"criteria\":{\"type_ids\":[\"erratum\"],\"filters\":{\"unit\":{\"id\":\"$ERRATUM\"}}},\"override_config\":{\"recursive_conservative\":true,\"additional_repos\":{\"$BASE2\": \"$DEST\"}}}" \
         -H "Content-Type: application/json" \
         -X POST https://localhost/pulp/api/v2/repositories/$DEST/actions/associate/)
TASK_URL=$(echo $TASK_OUTPUT | jq -r '.spawned_tasks[]._href')
echo "Task url : " $TASK_URL
if [ -z "$TASK_URL" ]; then echo "NO TASK?!?"; exit; fi
wait_until_task_finished https://localhost$TASK_URL
pulp-admin rpm repo publish run --repo-id $DEST
"""

Comment 3 Partha Aji 2020-02-07 17:06:00 UTC
From a sat perpective -
This is  a non issue for regular publishes/publsih with depsolve. Its more a case of investigating if incremental update will hit this and fix. It is possible that cdn adds a new module with module default and if we say incremental update this errata that points to the new stuff  we may hit into issue.                                                                                                                                  
Will be good to have this fix in any case from pulp.

Comment 4 pulp-infra@redhat.com 2020-02-07 17:31:55 UTC
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.

Comment 5 pulp-infra@redhat.com 2020-02-07 17:31:57 UTC
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

Comment 6 pulp-infra@redhat.com 2020-02-11 15:43:09 UTC
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

Comment 7 pulp-infra@redhat.com 2020-02-11 16:02:26 UTC
All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.

Comment 9 pulp-infra@redhat.com 2020-02-27 17:02:31 UTC
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.

Comment 10 pulp-infra@redhat.com 2020-03-05 18:47:54 UTC
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Comment 12 Lai 2020-03-30 21:42:51 UTC
Step to test:

1. ssh into sat
2. install pulp-admin
3. run script:

export BASE1='foo'
export BASE2='bar'
export REMOTE1='https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-with-modules-modified/'
export REMOTE2='https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-unsigned-modified/'
export DEST1='destination1'
export DEST2='destination2'
export ERRATUM='RHEA-2012:0059'
pulp-admin rpm repo create --repo-id=$BASE1 --relative-url=$BASE1 --feed=$REMOTE1
pulp-admin rpm repo sync run --repo-id=$BASE1
pulp-admin rpm repo create --repo-id=$BASE2 --relative-url=$BASE2 --feed=$REMOTE2
pulp-admin rpm repo sync run --repo-id=$BASE2
pulp-admin rpm repo create --repo-id=$DEST1
pulp-admin rpm repo create --repo-id=$DEST2
TASK_OUTPUT=$(curl -k -u admin:admin --cert ~/.pulp/user-cert.pem \
         -d "{\"source_repo_id\":\"$BASE1\",\"criteria\":{\"type_ids\":[\"erratum\"],\"filters\":{\"unit\":{\"id\":\"$ERRATUM\"}}},\"override_config\":{\"recursive_conservative\":true,\"additional_repos\":{\"$BASE2\": \"$DEST2\"}}}" \
         -H "Content-Type: application/json" \
         -X POST https://localhost/pulp/api/v2/repositories/$DEST1/actions/associate/)
TASK_URL=$(echo $TASK_OUTPUT | jq -r '.spawned_tasks[]._href')
echo "Task url : " $TASK_URL
if [ -z "$TASK_URL" ]; then echo "NO TASK?!?"; exit; fi
wait_until_task_finished https://localhost$TASK_URL
pulp-admin rpm repo publish run --repo-id $DEST1
pulp-admin rpm repo publish run --repo-id $DEST2

4. check the log via /var/log/messages and see that there's no errors.
5. cat out  /var/lib/pulp/published/yum/https/repos/destination1/repodata/*modules.yaml*

4)
Expected:
Mar 30 17:35:41 dhcp-2-51 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[3daddb3e-5ee9-4dfd-b6eb-259f162ca01b]
Mar 30 17:35:41 dhcp-2-51 pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.publish.publish[bdd7e9e1-e7b7-4630-bb06-f75608315c02]
Mar 30 17:35:41 dhcp-2-51 pulp: celery.app.trace:INFO: [3daddb3e] Task pulp.server.async.tasks._queue_reserved_task[3daddb3e-5ee9-4dfd-b6eb-259f162ca01b] succeeded in 0.0318934479728s: None
Mar 30 17:35:41 dhcp-2-51 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[c0029bd4-b539-48d2-afc8-083216ea45f9]
Mar 30 17:35:41 dhcp-2-51 pulp: celery.app.trace:INFO: [bdd7e9e1] Task pulp.server.managers.repo.publish.publish[bdd7e9e1-e7b7-4630-bb06-f75608315c02] succeeded in 0.233778246038s: {'exception': None, 'repo_id': 'destination1', 'traceback': None, 'started': '2020-03-30T21:35:41Z', '_ns': 'repo_publish_results', 'completed': datetime.datetime(2020, 3, 30, 21, 35, 41, 898107, tzinfo=<isodate.tzinfo.Utc object at 0x7f03b2f7ac50>), 'error_message': None, 'distributor_type_id': 'yum_distributor', 'distributor_id': 'yum_distributor', 'summary': {'generate sqlite': 'SKIPPED', 'initialize_repo_metadata': 'FINISHED', 'remove_old_repodata': 'FINISHED', 'rpms': 'FINISHED', 'modules': 'FINISHED', 'close_repo_metadata': 'FINISHED', 'drpms': 'SKIPPED', 'comps': 'FINISHED', 'distribution': 'FINISHED', 'repoview': 'SKIPPED', 'publish_directory': 'FINISHED', 'errata': 'FINISHED', 'metadata': 'FINISHED'}, 'result': 'success', 'id': '5e82662d067da34ed6b30328', 'details': [{'num_processed': 1, 'items_total': 1, 'state': 'FINISHED', 'num_success': 1, 'error_details': [...], 'description': 'Initializing repo metadata', 'num_failures': 0, 'step_id': '76ddf5ea-6e35-440e-af7e-5932d8dd9ca9', 'step_type': 'initial...', ...}]}
Mar 30 17:35:41 dhcp-2-51 pulp: celery.app.trace:INFO: [c0029bd4] Task pulp.server.async.tasks._release_resource[c0029bd4-b539-48d2-afc8-083216ea45f9] succeeded in 0.00185615505325s: None
Mar 30 17:35:44 dhcp-2-51 dhclient[1664]: DHCPREQUEST on eth0 to 10.11.5.19 port 67 (xid=0x171c8c16)
Mar 30 17:35:46 dhcp-2-51 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[aa55d767-84cb-4f79-bd31-f5967a151a8b]
Mar 30 17:35:46 dhcp-2-51 pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.publish.publish[3e0d777f-f071-43b1-9b1f-b5c1a42862fd]
Mar 30 17:35:46 dhcp-2-51 pulp: celery.app.trace:INFO: [aa55d767] Task pulp.server.async.tasks._queue_reserved_task[aa55d767-84cb-4f79-bd31-f5967a151a8b] succeeded in 0.0274146299926s: None
Mar 30 17:35:46 dhcp-2-51 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[cc7a8de5-d663-4a4e-8f0f-34d35d969600]
Mar 30 17:35:46 dhcp-2-51 pulp: celery.app.trace:INFO: [3e0d777f] Task pulp.server.managers.repo.publish.publish[3e0d777f-f071-43b1-9b1f-b5c1a42862fd] succeeded in 0.163032072014s: {'exception': None, 'repo_id': 'destination2', 'traceback': None, 'started': '2020-03-30T21:35:46Z', '_ns': 'repo_publish_results', 'completed': datetime.datetime(2020, 3, 30, 21, 35, 46, 735665, tzinfo=<isodate.tzinfo.Utc object at 0x7f03b2f7ac50>), 'error_message': None, 'distributor_type_id': 'yum_distributor', 'distributor_id': 'yum_distributor', 'summary': {'generate sqlite': 'SKIPPED', 'initialize_repo_metadata': 'FINISHED', 'remove_old_repodata': 'FINISHED', 'rpms': 'FINISHED', 'modules': 'SKIPPED', 'close_repo_metadata': 'FINISHED', 'drpms': 'SKIPPED', 'comps': 'FINISHED', 'distribution': 'FINISHED', 'repoview': 'SKIPPED', 'publish_directory': 'FINISHED', 'errata': 'FINISHED', 'metadata': 'FINISHED'}, 'result': 'success', 'id': '5e826632067da34ed6b30329', 'details': [{'num_processed': 1, 'items_total': 1, 'state': 'FINISHED', 'num_success': 1, 'error_details': [...], 'description': 'Initializing repo metadata', 'num_failures': 0, 'step_id': 'e9192bac-823f-402e-82f3-3ce744605501', 'step_type': 'initiali...', ...}]}
Mar 30 17:35:46 dhcp-2-51 pulp: celery.app.trace:INFO: [cc7a8de5] Task pulp.server.async.tasks._release_resource[cc7a8de5-d663-4a4e-8f0f-34d35d969600] succeeded in 0.00176209799247s: None
Mar 30 17:36:03 dhcp-2-51 dhclient[1664]: DHCPREQUEST on eth0 to 10.11.5.19 port 67 (xid=0x171c8c16)

Actual:
Mar 30 17:35:41 dhcp-2-51 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[3daddb3e-5ee9-4dfd-b6eb-259f162ca01b]
Mar 30 17:35:41 dhcp-2-51 pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.publish.publish[bdd7e9e1-e7b7-4630-bb06-f75608315c02]
Mar 30 17:35:41 dhcp-2-51 pulp: celery.app.trace:INFO: [3daddb3e] Task pulp.server.async.tasks._queue_reserved_task[3daddb3e-5ee9-4dfd-b6eb-259f162ca01b] succeeded in 0.0318934479728s: None
Mar 30 17:35:41 dhcp-2-51 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[c0029bd4-b539-48d2-afc8-083216ea45f9]
Mar 30 17:35:41 dhcp-2-51 pulp: celery.app.trace:INFO: [bdd7e9e1] Task pulp.server.managers.repo.publish.publish[bdd7e9e1-e7b7-4630-bb06-f75608315c02] succeeded in 0.233778246038s: {'exception': None, 'repo_id': 'destination1', 'traceback': None, 'started': '2020-03-30T21:35:41Z', '_ns': 'repo_publish_results', 'completed': datetime.datetime(2020, 3, 30, 21, 35, 41, 898107, tzinfo=<isodate.tzinfo.Utc object at 0x7f03b2f7ac50>), 'error_message': None, 'distributor_type_id': 'yum_distributor', 'distributor_id': 'yum_distributor', 'summary': {'generate sqlite': 'SKIPPED', 'initialize_repo_metadata': 'FINISHED', 'remove_old_repodata': 'FINISHED', 'rpms': 'FINISHED', 'modules': 'FINISHED', 'close_repo_metadata': 'FINISHED', 'drpms': 'SKIPPED', 'comps': 'FINISHED', 'distribution': 'FINISHED', 'repoview': 'SKIPPED', 'publish_directory': 'FINISHED', 'errata': 'FINISHED', 'metadata': 'FINISHED'}, 'result': 'success', 'id': '5e82662d067da34ed6b30328', 'details': [{'num_processed': 1, 'items_total': 1, 'state': 'FINISHED', 'num_success': 1, 'error_details': [...], 'description': 'Initializing repo metadata', 'num_failures': 0, 'step_id': '76ddf5ea-6e35-440e-af7e-5932d8dd9ca9', 'step_type': 'initial...', ...}]}
Mar 30 17:35:41 dhcp-2-51 pulp: celery.app.trace:INFO: [c0029bd4] Task pulp.server.async.tasks._release_resource[c0029bd4-b539-48d2-afc8-083216ea45f9] succeeded in 0.00185615505325s: None
Mar 30 17:35:44 dhcp-2-51 dhclient[1664]: DHCPREQUEST on eth0 to 10.11.5.19 port 67 (xid=0x171c8c16)
Mar 30 17:35:46 dhcp-2-51 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[aa55d767-84cb-4f79-bd31-f5967a151a8b]
Mar 30 17:35:46 dhcp-2-51 pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.publish.publish[3e0d777f-f071-43b1-9b1f-b5c1a42862fd]
Mar 30 17:35:46 dhcp-2-51 pulp: celery.app.trace:INFO: [aa55d767] Task pulp.server.async.tasks._queue_reserved_task[aa55d767-84cb-4f79-bd31-f5967a151a8b] succeeded in 0.0274146299926s: None
Mar 30 17:35:46 dhcp-2-51 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[cc7a8de5-d663-4a4e-8f0f-34d35d969600]
Mar 30 17:35:46 dhcp-2-51 pulp: celery.app.trace:INFO: [3e0d777f] Task pulp.server.managers.repo.publish.publish[3e0d777f-f071-43b1-9b1f-b5c1a42862fd] succeeded in 0.163032072014s: {'exception': None, 'repo_id': 'destination2', 'traceback': None, 'started': '2020-03-30T21:35:46Z', '_ns': 'repo_publish_results', 'completed': datetime.datetime(2020, 3, 30, 21, 35, 46, 735665, tzinfo=<isodate.tzinfo.Utc object at 0x7f03b2f7ac50>), 'error_message': None, 'distributor_type_id': 'yum_distributor', 'distributor_id': 'yum_distributor', 'summary': {'generate sqlite': 'SKIPPED', 'initialize_repo_metadata': 'FINISHED', 'remove_old_repodata': 'FINISHED', 'rpms': 'FINISHED', 'modules': 'SKIPPED', 'close_repo_metadata': 'FINISHED', 'drpms': 'SKIPPED', 'comps': 'FINISHED', 'distribution': 'FINISHED', 'repoview': 'SKIPPED', 'publish_directory': 'FINISHED', 'errata': 'FINISHED', 'metadata': 'FINISHED'}, 'result': 'success', 'id': '5e826632067da34ed6b30329', 'details': [{'num_processed': 1, 'items_total': 1, 'state': 'FINISHED', 'num_success': 1, 'error_details': [...], 'description': 'Initializing repo metadata', 'num_failures': 0, 'step_id': 'e9192bac-823f-402e-82f3-3ce744605501', 'step_type': 'initiali...', ...}]}
Mar 30 17:35:46 dhcp-2-51 pulp: celery.app.trace:INFO: [cc7a8de5] Task pulp.server.async.tasks._release_resource[cc7a8de5-d663-4a4e-8f0f-34d35d969600] succeeded in 0.00176209799247s: None
Mar 30 17:36:03 dhcp-2-51 dhclient[1664]: DHCPREQUEST on eth0 to 10.11.5.19 port 67 (xid=0x171c8c16)


5)
Expected:
File should only contains duck and kangaroo

Actual:
File only contains duck and kangaroo

Marking issue as verified on 6.7.0_17

Comment 14 errata-xmlrpc 2020-04-14 13:28:43 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:1454