Bug 2033951

Description of problem:

If Alma Linux Base OS repo is one of the custom repos, then the "satellite-maintain content prepare" step will fail to complete the pulp2-pulp3 migration.


Version-Release number of selected component (if applicable):

Satellite 6.9 and 6.10

How reproducible:

Always

Steps to Reproduce:
1. Install Satellite 6.9

2. Have four almalinux repos created with the following URLs and then sync them.

# hammer repository list --product AlmaLinux --organization-id 1
---|-------------------------|-----------|--------------|-------------------------------------------------------------
ID | NAME                    | PRODUCT   | CONTENT TYPE | URL                                                         
---|-------------------------|-----------|--------------|-------------------------------------------------------------
1  | Alma Linux 8 AppStream  | AlmaLinux | yum          | https://repo.almalinux.org/almalinux/8/AppStream/x86_64/os/ 
2  | Alma Linux 8 Extras     | AlmaLinux | yum          | https://repo.almalinux.org/almalinux/8/extras/x86_64/os/    
3  | Alma Linux 8 OS         | AlmaLinux | yum          | https://repo.almalinux.org/almalinux/8/BaseOS/x86_64/os/    
4  | Alma Linux 8 PowerTools | AlmaLinux | yum          | https://repo.almalinux.org/almalinux/8/PowerTools/x86_64/os/
---|-------------------------|-----------|--------------|-------------------------------------------------------------


# echo "select name,url,download_policy,content_type,verify_ssl_on_sync from katello_root_repositories where name ilike '%Alma%';" | su - postgres -c "psql foreman"
          name           |                             url                              | download_policy | content_type | verify_ssl_on_sync 
-------------------------+--------------------------------------------------------------+-----------------+--------------+--------------------
 Alma Linux 8 AppStream  | https://repo.almalinux.org/almalinux/8/AppStream/x86_64/os/  | immediate       | yum          | t
 Alma Linux 8 Extras     | https://repo.almalinux.org/almalinux/8/extras/x86_64/os/     | immediate       | yum          | t
 Alma Linux 8 OS         | https://repo.almalinux.org/almalinux/8/BaseOS/x86_64/os/     | immediate       | yum          | t
 Alma Linux 8 PowerTools | https://repo.almalinux.org/almalinux/8/PowerTools/x86_64/os/ | immediate       | yum          | t
(4 rows)

3. Once the sync completes, Add them to a CV and publish the CV

4. Try content migration i.e.

# satellite-maintain prep-6.10-upgrade

# satellite-maintain content prepare


Actual results:


* The "content prepare" step will fail.

2021-12-18 22:49:08 +0530: Migrating rpm content to Pulp 3 17755/17839
2021-12-18 22:49:18 +0530: Migrating distribution content to Pulp 3 0/5
2021-12-18 22:49:28 +0530: Migrating distribution content to Pulp 3 0/5
2021-12-18 22:49:38 +0530: Migrating distribution content to Pulp 3 3/5
2021-12-18 22:49:48 +0530: Migrating erratum content to Pulp 3 0/3868
2021-12-18 22:49:58 +0530: Migrating erratum content to Pulp 3 0/3868
2021-12-18 22:50:08 +0530: Migrating modulemd content to Pulp 3 0/108Migration failed, You will want to investigate: https://satellite.example.com/foreman_tasks/tasks/afa88284-f2e6-409c-898c-96a7021fc35a
rake aborted!
ForemanTasks::TaskError: Task afa88284-f2e6-409c-898c-96a7021fc35a: Katello::Errors::Pulp3Error: No declared artifact with relative path "images/boot.iso" for content "<DistributionTree: pk=c21cf106-dc3b-4615-8541-229726d4a06d>"
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.46/lib/katello/tasks/pulp3_migration.rake:43:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
                                                                      [FAIL]
Failed executing preserve_output=true foreman-rake katello:pulp3_migration, exit status 1
--------------------------------------------------------------------------------
Scenario [Prepare content for Pulp 3] failed.

The following steps ended up in failing state:

  [content-prepare]


* Investigation shows the DistributionTree primary key is pointing to a treeinfo file of Alma Linux repo.


# echo "select * from rpm_distributiontree where content_ptr_id = 'c21cf106-dc3b-4615-8541-229726d4a06d';" | su - postgres -c "psql pulpcore"
            content_ptr_id            | header_version | release_name | release_short | release_version | release_is_layered | base_product_name | base_product_short | base_product_version |  arch  | build_timestamp | instimage | mainimage | discnum | totaldiscs 
--------------------------------------+----------------+--------------+---------------+-----------------+--------------------+-------------------+--------------------+----------------------+--------+-----------------+-----------+-----------+---------+------------
 c21cf106-dc3b-4615-8541-229726d4a06d | 1.2            | AlmaLinux    | AlmaLinux     | 8               | f                  |                   |                    |                      | x86_64 |      1636649716 |           |           |         |           
(1 row)


# echo "select * from core_contentartifact where content_id = 'c21cf106-dc3b-4615-8541-229726d4a06d';" | su - postgres -c "psql pulpcore"
               pulp_id                |           pulp_created           |        pulp_last_updated         | relative_path |             artifact_id              |              content_id              
--------------------------------------+----------------------------------+----------------------------------+---------------+--------------------------------------+--------------------------------------
 cfb23e31-43d2-4332-80ec-5d391c49c98d | 2021-12-18 22:49:52.015523+05:30 | 2021-12-18 22:49:52.015541+05:30 | .treeinfo     | 5258ea04-d769-4d7a-b003-ae4c5f47ac49 | c21cf106-dc3b-4615-8541-229726d4a06d
(1 row)


# echo "select * from core_artifact where pulp_id = 'cfb23e31-43d2-4332-80ec-5d391c49c98d';" | su - postgres -c "psql pulpcore"
 pulp_id | pulp_created | pulp_last_updated | file | size | md5 | sha1 | sha224 | sha256 | sha384 | sha512 
---------+--------------+-------------------+------+------+-----+------+--------+--------+--------+--------
(0 rows)



Expected results:

The migration process should be able to handle the migration of Almalinux repos as well and complete the same without any errors.


Additional info:

It seems to be similar to https://bugzilla.redhat.com/show_bug.cgi?id=1945377  and https://bugzilla.redhat.com/show_bug.cgi?id=1964530 which were for RHEL 8 and CentOS 8 Base OS.

I have checked the .treeinfo file for Base OS repo of Alma Linux product that was synced. It has valid path for all files.

[images-x86_64]
boot.iso = images/boot.iso
efiboot.img = images/efiboot.img
initrd = images/pxeboot/initrd.img
kernel = images/pxeboot/vmlinuz


# pwd
/var/lib/pulp/published/yum/https/repos/RedHat/Library/custom/AlmaLinux/Alma_Linux_8_OS

# ll images/ -R
images/:
total 0
lrwxrwxrwx. 1 apache apache 122 Dec 18 21:05 boot.iso -> /var/lib/pulp/content/units/distribution/97/e022aa9f45c173f3b2dfcc6db4104eadb1ea84d7ed6dc82fb93e28d78d4da4/images/boot.iso
lrwxrwxrwx. 1 apache apache 125 Dec 18 21:05 efiboot.img -> /var/lib/pulp/content/units/distribution/97/e022aa9f45c173f3b2dfcc6db4104eadb1ea84d7ed6dc82fb93e28d78d4da4/images/efiboot.img
lrwxrwxrwx. 1 apache apache 125 Dec 18 21:05 install.img -> /var/lib/pulp/content/units/distribution/97/e022aa9f45c173f3b2dfcc6db4104eadb1ea84d7ed6dc82fb93e28d78d4da4/images/install.img
drwxrwx---. 2 apache apache  39 Dec 18 21:05 pxeboot

images/pxeboot:
total 0
lrwxrwxrwx. 1 apache apache 132 Dec 18 21:05 initrd.img -> /var/lib/pulp/content/units/distribution/97/e022aa9f45c173f3b2dfcc6db4104eadb1ea84d7ed6dc82fb93e28d78d4da4/images/pxeboot/initrd.img
lrwxrwxrwx. 1 apache apache 129 Dec 18 21:05 vmlinuz -> /var/lib/pulp/content/units/distribution/97/e022aa9f45c173f3b2dfcc6db4104eadb1ea84d7ed6dc82fb93e28d78d4da4/images/pxeboot/vmlinuz


# for i in images/*; do echo Checking $i : `test -e $i && echo "Symlink OK" || echo "Symlink Broken"` ; done
Checking images/boot.iso : Symlink OK
Checking images/efiboot.img : Symlink OK
Checking images/install.img : Symlink OK
Checking images/pxeboot : Symlink OK

# for i in images/pxeboot/*; do echo Checking $i : `test -e $i && echo "Symlink OK" || echo "Symlink Broken"` ; done
Checking images/pxeboot/initrd.img : Symlink OK
Checking images/pxeboot/vmlinuz : Symlink OK


NOTE: There is no workaround to get past this problem without deleting the repo and restarting the migration which is not a very good solution.