Bug 1939090

Summary: "Erratum record has migrated_pulp3_href NULL value" error message when running pulp_3_content_switchover
Product: Red Hat Satellite Reporter: Lai <ltran>
Component: PulpAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED ERRATA QA Contact: Lai <ltran>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.9.0CC: bmbouter, ggainey, ipanova, jsherril, mmccune, pcreech, rchan, ttereshc, zhunting
Target Milestone: 6.9.0Keywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: python3-pulp_2to3_migration-0.10.0 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-04-21 13:26:20 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Lai 2021-03-15 15:48:38 UTC
Description of problem:
Running switchover command produces error message "ERROR: at least one Erratum record has migrated_pulp3_href NULL value"

Version-Release number of selected component (if applicable):
6.9.0 snap 17

How reproducible:
100%

Steps to Reproduce:
1. Sync a few RH repos with erratas (I synced Ansible Engine 2.9 RPMs, Extended Update Support RPMs, and Collections RPMs for Red Hat Enterprise Linux 7 Server)
2. Create a cv and add repos from #1
3. Publish cv
4. Run migration command until it's finished
5. Remove a repo that contains erratas from cv (I did Collections RPMs for Red Hat Enterprise Linux 7)
6. Republish cv
7. Run Migration command until it's finished
8. Run the two commands below for switchover:

# systemctl start pulpcore-resource-manager pulpcore-api pulpcore-content  pulpcore-worker@1 pulpcore-worker@2  pulpcore-worker@3 pulpcore-worker@4
# systemctl enable pulpcore-resource-manager pulpcore-api pulpcore-content  pulpcore-worker@1 pulpcore-worker@2  pulpcore-worker@3 pulpcore-worker@4

# foreman-rake katello:pulp3_content_switchover

Expected Result
Switchover should complete successfully without issues.

Actual result:
Description of problem:
Running switchover command produces error message "ERROR: at least one Erratum record has migrated_pulp3_href NULL value"


You can check dhcp-2-123.vms.sat.rdu2.redhat.com as a reproducer.

Comment 1 Tanya Tereshchenko 2021-03-15 15:57:17 UTC
Lai, can you confirm that if you perform a switchover after step 4, there are no issues.

Comment 2 Tanya Tereshchenko 2021-03-15 16:26:19 UTC
Justin, I don't seem to find any pulp2content without the pulp3 counterpart.
Could you look at the reproducer which Lai provided or give me an idea what triggers that Katello error message?
Thanks

pulpcore=> select count(*) from pulp_2to3_migration_pulp2content where pulp2_content_type_id='erratum' and pulp3_content_id is null;
 count 
-------
     0
(1 row)

pulpcore=> select count(*) from pulp_2to3_migration_pulp2content where pulp3_content_id is null;
 count 
-------
     0
(1 row)

Comment 3 Grant Gainey 2021-03-15 18:51:29 UTC
(In reply to Lai from comment #0)
> Description of problem:
> Running switchover command produces error message "ERROR: at least one
> Erratum record has migrated_pulp3_href NULL value"

migrated_pulp3_href isn't a Pulp3 string, anywhere. I'm guessing this is katello saying "I found an erratum and I don't know where it came from".

Which makes sense? Because the reproducer is "migrate a thing-with-an-erratum, then delete that thing, then migrate again"? So erratum (whatever) is "in Pulp3", with a CV that points to it, and then it's not part of the next migration. What does katello expect will happen in this case? (for that matter, what does anyone expect?)

Comment 4 Lai 2021-03-15 19:53:17 UTC
Hey Tanya,

Regular switchover after migration from step 4 works as expected.  There is no issues from that.

Comment 5 Justin Sherrill 2021-03-16 13:46:55 UTC
To add some more information, it appears that errata that are already migrated, but appear in new repositories in a subsequent migration are not being migrated to the new repositories. 

Here's steps to reproduce with:

1.  make a repository A with at least 1 erratum
2.  make a 2nd repository B, copy all content from A to B
3.  migrate both repositories, here's what my migration plan looks like (there's technically a 3rd repo we are just using for the reproducer):

{
   "plugins":[
      {
         "type":"docker",
         "repositories":[
            
         ]
      },
      {
         "type":"iso",
         "repositories":[
            
         ]
      },
      {
         "type":"rpm",
         "repositories":[
            {
               "name":"f86e5a95-67c1-4726-8ed4-d1eca9130f43",
               "repository_versions":[
                  {
                     "pulp2_repository_id":"f86e5a95-67c1-4726-8ed4-d1eca9130f43",
                     "pulp2_distributor_repository_ids":[
                        "f86e5a95-67c1-4726-8ed4-d1eca9130f43"
                     ]
                  }
               ],
               "pulp2_importer_repository_id":"f86e5a95-67c1-4726-8ed4-d1eca9130f43"
            },
            {
               "name":"mycontentview-need",
               "repository_versions":[
                  {
                     "pulp2_repository_id":"1-mycontentview-v2_0-f86e5a95-67c1-4726-8ed4-d1eca9130f43",
                     "pulp2_distributor_repository_ids":[
                        "1-mycontentview-Library-f86e5a95-67c1-4726-8ed4-d1eca9130f43",
                        "1-mycontentview-v2_0-f86e5a95-67c1-4726-8ed4-d1eca9130f43"
                     ]
                  }
               ]
            }
         ]
      }
   ]
}

4.  Confirm that Two repositories (and versions) get created in pulp3, both having errata
5.  Create a new pulp2 repository C and copy all content from A to C
6.  Re-migrate, with this new migration plan:


{
   "plugins":[
      {
         "type":"docker",
         "repositories":[
            
         ]
      },
      {
         "type":"iso",
         "repositories":[
            
         ]
      },
      {
         "type":"rpm",
         "repositories":[
            {
               "name":"f86e5a95-67c1-4726-8ed4-d1eca9130f43",
               "repository_versions":[
                  {
                     "pulp2_repository_id":"f86e5a95-67c1-4726-8ed4-d1eca9130f43",
                     "pulp2_distributor_repository_ids":[
                        "f86e5a95-67c1-4726-8ed4-d1eca9130f43"
                     ]
                  }
               ],
               "pulp2_importer_repository_id":"f86e5a95-67c1-4726-8ed4-d1eca9130f43"
            },
            {
               "name":"mycontentview-need",
               "repository_versions":[
                  {
                     "pulp2_repository_id":"1-mycontentview-v2_0-f86e5a95-67c1-4726-8ed4-d1eca9130f43",
                     "pulp2_distributor_repository_ids":[
                        "1-mycontentview-v2_0-f86e5a95-67c1-4726-8ed4-d1eca9130f43"
                     ]
                  },
                  {
                     "pulp2_repository_id":"1-mycontentview-v3_0-f86e5a95-67c1-4726-8ed4-d1eca9130f43",
                     "pulp2_distributor_repository_ids":[
                        "1-mycontentview-Library-f86e5a95-67c1-4726-8ed4-d1eca9130f43",
                        "1-mycontentview-v3_0-f86e5a95-67c1-4726-8ed4-d1eca9130f43"
                     ]
                  }
               ]
            }
         ]
      }
   ]
}



After migrating, check the content in the newly created repo version for 1-mycontentview-v3_0-f86e5a95-67c1-4726-8ed4-d1eca9130f43, it will contain no advisories (errata)

Comment 6 pulp-infra@redhat.com 2021-03-17 17:26:23 UTC
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.

Comment 7 pulp-infra@redhat.com 2021-03-17 17:26:25 UTC
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.

Comment 8 pulp-infra@redhat.com 2021-03-22 14:06:55 UTC
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.

Comment 9 pulp-infra@redhat.com 2021-03-22 15:06:41 UTC
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

Comment 10 pulp-infra@redhat.com 2021-03-22 16:10:55 UTC
All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.

Comment 11 pulp-infra@redhat.com 2021-03-23 19:06:21 UTC
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Comment 12 Lai 2021-03-30 17:26:16 UTC
Steps to retest

Steps to Reproduce:
1. Sync a repo with at least one errata (I synced Ansible Engine 2.9 RPMs)
2. Create a cv and add repo from #1
3. Publish cv
4. Run migration command until it's finished
5. Remove a repo that contains erratas from cv
6. Republish cv
7. Run Migration command until it's finished
8. Run the two commands below for switchover:

# systemctl start pulpcore-resource-manager pulpcore-api pulpcore-content  pulpcore-worker@1 pulpcore-worker@2  pulpcore-worker@3 pulpcore-worker@4
# systemctl enable pulpcore-resource-manager pulpcore-api pulpcore-content  pulpcore-worker@1 pulpcore-worker@2  pulpcore-worker@3 pulpcore-worker@4

# foreman-rake katello:pulp3_content_switchover

Expected Result
Switchover should complete successfully without issues.

Actual result:
Switchover completed successfully without issues.  Ran rest of migration steps and it was success as well.

Verified on 6.9.0_019.1 with python3-pulp-2to3-migration-0.10.0-1.el7pc.noarch

Comment 15 errata-xmlrpc 2021-04-21 13:26:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Satellite 6.9 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:1313