Bug 2133468

Summary: Upgrade fails to apply rpm.0044_noartifact_modules pulpcore migration
Product: Red Hat Satellite Reporter: Lukas Pramuk <lpramuk>
Component: PulpAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED ERRATA QA Contact: Lukas Pramuk <lpramuk>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.12.0CC: dalley, dkliban, ggainey, lvrtelov, pcreech, ppicka, rchan
Target Milestone: 6.12.0Keywords: Triaged, UpgradeBlocker, Upgrades
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: python-pulp-rpm-3.18.7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-11-16 13:35:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Lukas Pramuk 2022-10-10 14:55:03 UTC
Description of problem:
Upgrade to 6.12.0 Snap14 fails during installer run since pulpcore-manager fails to apply rpm.0044_noartifact_modules migration

Version-Release number of selected component (if applicable):
6.12.0 Snap14
python39-pulp-rpm-3.18.3-1.el8pc.noarch

How reproducible:
100% (when there is reasonable content)

Steps to Reproduce:
1. Upgrade from 6.11.z to 6.12.0

# satellite-maintain upgrade run --whitelist="disk-performance, repositories-validate, repositories-setup" --target-version 6.12 -y
Running Migration scripts to Satellite 6.12
================================================================================
Setup repositories:                                                   [SKIPPED]
--------------------------------------------------------------------------------
Unlock packages:                                                      [OK]
--------------------------------------------------------------------------------
Update package(s) :                                                   [OK]
--------------------------------------------------------------------------------
Procedures::Installer::Upgrade:                                       [FAIL]
Failed executing LANG=en_US.utf-8 satellite-installer  --disable-system-checks, exit status 6:
...
2022-10-07 11:06:26 [ERROR ] [configure] 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
2022-10-07 11:06:26 [ERROR ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: change from 'notrun' to ['0'] failed: 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
2022-10-07 11:06:36 [ERROR ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]: Failed to call refresh: 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
2022-10-07 11:06:36 [ERROR ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]: 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]


Actual results:
upgrade fails

Expected results:
upgrade succeeds


Additional info:
2022-10-07 12:13:26 [DEBUG ] [configure] Exec[pulpcore-manager migrate --noinput](provider=posix): Executing 'pulpcore-manager migrate --noinput'
2022-10-07 12:13:26 [DEBUG ] [configure] Executing with uid=pulp: 'pulpcore-manager migrate --noinput'
2022-10-07 12:13:32 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:
System check identified some issues:

WARNINGS:
?: (guardian.W001) Guardian authentication backend is not hooked. You can add this in settings as eg: `AUTHENTICATION_BACKENDS = ('django.contrib.auth.backends.ModelBackend', 'guardian.backends.ObjectPermissionBackend')`.
Operations to perform:
  Apply all migrations: admin, ansible, auth, certguard, container, contenttypes, core, file, guardian, rpm, sessions
Running migrations:
  Applying rpm.0044_noartifact_modules...Traceback (most recent call last):
  File "/usr/bin/pulpcore-manager", line 33, in <module>
    sys.exit(load_entry_point('pulpcore==3.18.10', 'console_scripts', 'pulpcore-manager')())
  File "/usr/lib/python3.9/site-packages/pulpcore/app/manage.py", line 11, in manage
    execute_from_command_line(sys.argv)
  File "/usr/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/usr/lib/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/usr/lib/python3.9/site-packages/django/core/management/base.py", line 89, in wrapped
    res = handle_func(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/django/core/management/commands/migrate.py", line 244, in handle
    post_migrate_state = executor.migrate(
  File "/usr/lib/python3.9/site-packages/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/usr/lib/python3.9/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/lib/python3.9/site-packages/django/db/migrations/executor.py", line 227, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/usr/lib/python3.9/site-packages/django/db/migrations/migration.py", line 126, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/usr/lib/python3.9/site-packages/django/db/migrations/operations/special.py", line 190, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/usr/lib/python3.9/site-packages/pulp_rpm/app/migrations/0044_noartifact_modules.py", line 21, in convert_artifact_to_snippets
    content_artifact = ContentArtifact.objects.get(artifact__pk=artifact.pk)
  File "/usr/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/django/db/models/query.py", line 439, in get
    raise self.model.MultipleObjectsReturned(
__fake__.MultipleObjectsReturned: get() returned more than one ContentArtifact -- it returned 2!

Comment 1 Daniel Alley 2022-10-10 18:16:42 UTC
If you apply this patch, does the migration work? 


Untitled
From WebUI, 25 Minutes ago, written in Diff-output.
This paste will explode in 23 Hours.	
URL https://paste.centos.org/view/6bc966bb
Embed Show code
Download Paste or View Raw

        diff --git a/pulp_rpm/app/migrations/0044_noartifact_modules.py b/pulp_rpm/app/migrations/0044_noartifact_modules.py
        index 7bedf12d..6e65976e 100644
        --- a/pulp_rpm/app/migrations/0044_noartifact_modules.py
        +++ b/pulp_rpm/app/migrations/0044_noartifact_modules.py
        @@ -18,7 +18,7 @@ def convert_artifact_to_snippets(apps, schema_editor):
                 for module in Modulemd.objects.all():
                     artifact = module._artifacts.get()
                     module.snippet = artifact.file.read().decode("utf-8")
        -            content_artifact = ContentArtifact.objects.get(artifact__pk=artifact.pk)
        +            content_artifact = ContentArtifact.objects.get(artifact__pk=artifact.pk, content__pk=module.pk)
                     content_artifacts_to_delete.append(content_artifact.pk)
                     artifacts_to_delete.append(artifact.pk)
                     modules_with_snippet.append(module)
        @@ -26,7 +26,7 @@ def convert_artifact_to_snippets(apps, schema_editor):
                 for default in ModulemdDefaults.objects.all():
                     artifact = default._artifacts.get()
                     default.snippet = artifact.file.read().decode("utf-8")
        -            content_artifact = ContentArtifact.objects.get(artifact__pk=artifact.pk)
        +            content_artifact = ContentArtifact.objects.get(artifact__pk=artifact.pk, content__pk=default.pk)
                     content_artifacts_to_delete.append(content_artifact.pk)
                     artifacts_to_delete.append(artifact.pk)
                     defaults_with_snippet.append(default)

Comment 3 Lukas Pramuk 2022-10-11 08:57:19 UTC
The proposed patch does not work

# wget -O- https://paste.centos.org/view/download/6bc966bb |  patch -N -p1 -d /usr/lib/python3.9/site-packages
# su - pulp -s /bin/bash
$ PULP_SETTINGS=/etc/pulp/settings.py pulpcore-manager migrate
System check identified some issues:

WARNINGS:
?: (guardian.W001) Guardian authentication backend is not hooked. You can add this in settings as eg: `AUTHENTICATION_BACKENDS = ('django.contrib.auth.backends.ModelBackend', 'guardian.backends.ObjectPermissionBackend')`.
Operations to perform:
  Apply all migrations: admin, ansible, auth, certguard, container, contenttypes, core, file, guardian, rpm, sessions
Running migrations:
  Applying rpm.0044_noartifact_modules...Traceback (most recent call last):
  File "/usr/bin/pulpcore-manager", line 33, in <module>
    sys.exit(load_entry_point('pulpcore==3.18.10', 'console_scripts', 'pulpcore-manager')())
  File "/usr/lib/python3.9/site-packages/pulpcore/app/manage.py", line 11, in manage
    execute_from_command_line(sys.argv)
  File "/usr/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/usr/lib/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/usr/lib/python3.9/site-packages/django/core/management/base.py", line 89, in wrapped
    res = handle_func(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/django/core/management/commands/migrate.py", line 244, in handle
    post_migrate_state = executor.migrate(
  File "/usr/lib/python3.9/site-packages/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/usr/lib/python3.9/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/lib/python3.9/site-packages/django/db/migrations/executor.py", line 227, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/usr/lib/python3.9/site-packages/django/db/migrations/migration.py", line 126, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/usr/lib/python3.9/site-packages/django/db/migrations/operations/special.py", line 190, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/usr/lib/python3.9/site-packages/pulp_rpm/app/migrations/0044_noartifact_modules.py", line 45, in convert_artifact_to_snippets
    Artifact.objects.filter(pk__in=artifacts_to_delete).delete()
  File "/usr/lib/python3.9/site-packages/django/db/models/query.py", line 745, in delete
    collector.collect(del_query)
  File "/usr/lib/python3.9/site-packages/django/db/models/deletion.py", line 302, in collect
    raise ProtectedError(
django.db.models.deletion.ProtectedError: ("Cannot delete some instances of model 'Artifact' because they are referenced through protected foreign keys: 'ContentArtifact.artifact'.", {<ContentArtifact: ContentArtifact object (fed4acbe-e8a2-4a96-8aba-893d32e28fa7)>})

Comment 7 Lukas Pramuk 2022-10-14 09:31:02 UTC
VERIFIED.

@Satellite 6.12.0 Snap15
python39-pulp-rpm-3.18.7-1.el8pc.noarch

by using Upgrades Automation as the BZ was originally reproduced here:

upgrade-phase job with Satellite and Capsule scenario

>>> upgrade from 6.11.z to 6.12.0 passed successfully

Comment 11 errata-xmlrpc 2022-11-16 13:35:51 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.12 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:8506

Comment 12 pulp-infra@redhat.com 2022-11-16 14:06:40 UTC
The Pulp upstream bug status is at closed. Updating the external tracker on this bug.