Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2233448 - CV with filters (or republishing any CV version) causes modules lose flags ([d]efault)
Summary: CV with filters (or republishing any CV version) causes modules lose flags ([...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Pulp
Version: 6.13.2
Hardware: Unspecified
OS: Unspecified
unspecified
low
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: Satellite QE Team
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-08-22 09:02 UTC by Jan Jansky
Modified: 2024-05-15 01:54 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2024-05-15 01:54:46 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 36749 0 Normal New CV with filters (or republishing any CV version) causes modules lose flags ([d]efault) 2023-09-14 19:01:03 UTC
Red Hat Issue Tracker SAT-19777 0 None None None 2023-08-30 14:48:37 UTC
Red Hat Knowledge Base (Solution) 7061554 0 None None None 2024-03-25 13:38:48 UTC

Description Jan Jansky 2023-08-22 09:02:44 UTC 
Description of problem:
When content view version will be republished module list on consumer do not show [d] flags

Version-Release number of selected component (if applicable):
satellite-6.13.2-1.el8sat.noarch

How reproducible:
Always

Steps to Reproduce:
1. Publish new content view version
in my case containing below repositories
- Red Hat Satellite Client 6 for RHEL 9 x86_64 (RPMs)
- Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs)
- Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs)

2. Check 'dnf module list' on client
# dnf module list
Updating Subscription Management repositories.
Red Hat Satellite Client 6 for RHEL 9 x86_64 (RPMs)                                                                                                                                                           479  B/s | 3.8 kB     00:08    
Red Hat Satellite Client 6 for RHEL 9 x86_64 (RPMs)                                                                                                                                                           7.1 kB/s | 115 kB     00:16    
Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs)                                                                                                                                                         517  B/s | 4.1 kB     00:08    
Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs)                                                                                                                                                         847 kB/s |  13 MB     00:16    
Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs)                                                                                                                                                      574  B/s | 4.5 kB     00:08    
Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs)                                                                                                                                                      1.5 MB/s |  24 MB     00:16    
Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs)
Name                                       Stream                                 Profiles                                                              Summary                                                                               
maven                                      3.8                                    common [d]                                                            Java project management and project comprehension tool                                
nginx                                      1.22                                   common [d]                                                            nginx webserver                                                                       
nodejs                                     18                                     common [d], development, minimal, s2i                                 Javascript runtime                                                                    
php                                        8.1                                    common [d], devel, minimal                                            PHP scripting language                                                                
postgresql                                 15                                     client, server                                                        PostgreSQL server and client module                                                   
ruby                                       3.1                                    common [d]                                                            An interpreter of object-oriented scripting language  

3. Republish repository version
# hammer content-view version republish-repositories --id <id> --force true

4. Check 'dnf module list' on client
# dnf module list

Actual results:
# dnf module list
Updating Subscription Management repositories.
Red Hat Satellite Client 6 for RHEL 9 x86_64 (RPMs)                                                                                                                                                           288  B/s | 2.3 kB     00:08    
Red Hat Satellite Client 6 for RHEL 9 x86_64 (RPMs)                                                                                                                                                           7.2 kB/s | 116 kB     00:16    
Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs)                                                                                                                                                         333  B/s | 2.6 kB     00:08    
Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs)                                                                                                                                                         803 kB/s |  13 MB     00:16    
Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs)                                                                                                                                                      371  B/s | 2.9 kB     00:08    
Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs)                                                                                                                                                      1.3 MB/s |  20 MB     00:16    
Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs)
Name                                        Stream                                  Profiles                                                           Summary                                                                                
maven                                       3.8                                     common                                                             Java project management and project comprehension tool                                 
nginx                                       1.22                                    common                                                             nginx webserver                                                                        
nodejs                                      18                                      common, development, minimal, s2i                                  Javascript runtime                                                                     
php                                         8.1                                     common, devel, minimal                                             PHP scripting language                                                                 
postgresql                                  15                                      client, server                                                     PostgreSQL server and client module                                                    
ruby                                        3.1                                     common                                                             An interpreter of object-oriented scripting language                                   

Hint: [d]efault, [e]nabled, [x]disabled, [i]nstalled

Expected results:

# dnf module list
Updating Subscription Management repositories.
Red Hat Satellite Client 6 for RHEL 9 x86_64 (RPMs)                                                                                                                                                           479  B/s | 3.8 kB     00:08    
Red Hat Satellite Client 6 for RHEL 9 x86_64 (RPMs)                                                                                                                                                           7.1 kB/s | 115 kB     00:16    
Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs)                                                                                                                                                         517  B/s | 4.1 kB     00:08    
Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs)                                                                                                                                                         847 kB/s |  13 MB     00:16    
Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs)                                                                                                                                                      574  B/s | 4.5 kB     00:08    
Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs)                                                                                                                                                      1.5 MB/s |  24 MB     00:16    
Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs)
Name                                       Stream                                 Profiles                                                              Summary                                                                               
maven                                      3.8                                    common [d]                                                            Java project management and project comprehension tool                                
nginx                                      1.22                                   common [d]                                                            nginx webserver                                                                       
nodejs                                     18                                     common [d], development, minimal, s2i                                 Javascript runtime                                                                    
php                                        8.1                                    common [d], devel, minimal                                            PHP scripting language                                                                
postgresql                                 15                                     client, server                                                        PostgreSQL server and client module                                                   
ruby                                       3.1                                    common [d]                                                            An interpreter of object-oriented scripting language                                  

Hint: [d]efault, [e]nabled, [x]disabled, [i]nstalled

Additional info:
Comment 3 Pavel Moravec 2023-08-29 09:37:05 UTC 
A different reproducer for the same root cause suggests two things:
1) even for a CV without filters, republish CV version does expensively generate pieces of metadata *like* the CV has filters - while originally CV publish just quickly copied metadata from Library repo (is this expensive metadata generation really required? can't we behave like during CV publish here?)
2) publishing CV with filters does also forget to add the default modularity enablednes!

The reproducer itself:

- create CV with RHEL9 BaseOS and AppStream repos
- publish the CV and check modulemd-defaults in modules.yaml repodata (will be present)
- add CV filter "exclude sos RPMs". Those packages from BaseOS should have zero impact to AppStream repo content, at all (no AppStream content depends in either way on sos, imho)
- publish the CV and check modulemd-defaults in modules.yaml repodata (will be missing)
- replace the filter by "exclude acpid RPMs" (acpid is some AppStream RPM that should not (afaik, worth to double-check) impact anything modularity-related
- publish the CV again, and check the modulemd-defaults in modules.yaml repodata again (will be missing)

How to check modulemd-defaults in modules.yaml repodata : either using a client system assigned to the CV, or directly in artifacts bits on Satellite, via:

# cat modules-artifact.py
from pulp_rpm.app.models.repository import RpmDistribution

for distr in RpmDistribution.objects.all():
    if not distr.publication or not distr.publication.published_metadata:
        continue
    modfilter = distr.publication.published_metadata.filter(relative_path__contains='modules.yaml')
    if modfilter.count() > 0:
        print(f"{modfilter.first().contentartifact_set.first().artifact.file.path}   {distr.base_path}")
#

Then run:

cat modules-artifact.py | sudo -u pulp PULP_SETTINGS='/etc/pulp/settings.py' DJANGO_SETTINGS_MODULE='pulpcore.app.settings' pulpcore-manager shell | grep "/9/x86_64/appstream/os" | while read ART DPATH; do echo "$ART $(zgrep -c modulemd-defaults $ART) $DPATH"; done

to see:

..
/var/lib/pulp/media/artifact/28/972408412b083f2840c0b4fdecd2b273c44620bb462061e6f53656c28310ae 5 JJAN_ORG/content_views/RHEL9/4.0/content/dist/rhel9/9/x86_64/appstream/os
/var/lib/pulp/media/artifact/29/1042c6c4ef45ee07d9e2608bbd74d94a00861dd4a5438fcf2db160c1896ad8 0 JJAN_ORG/content_views/RHEL9/5.0/content/dist/rhel9/9/x86_64/appstream/os
..

meaning "RHEL9 content view in version 4.0 has modules.yaml artifact file stored in ..0ae file, and it has five modulemd-defaults pieces (the right modularity enabledness defaults)", BUT "RHEL9 content view in version 5.0 has modules.yaml artifact file stored in ..ad8 file, and it has zero modulemd-defaults pieces (missing modularity enabledness defaults)"


My gut feeling is that katello is guilty here (in Actions::Pulp3::Repository::CopyContent step? just guessing) - since CV filters (katello stuff) does affect the outcome; I *think* katello forgets to tell pulp "hey, please copy *also* modularity default enabledness info".
Comment 5 Ian Ballou 2023-08-30 14:46:03 UTC 
I agree with Pavel's analysis.

We probably need to copy all modulemd defaults in this method: https://github.com/Katello/katello/blob/master/app/services/katello/pulp3/repository/yum.rb#L381

For yum content in Sat 6.10, we used the "multi-copy" methods for all filtered content view copying. We switched over to only using the multi-copy methods when dependency solving to speed things up in Sat 6.11: https://github.com/Katello/katello/commit/3ac597399a06893457d6ceaa4ce339ad8aa44a4f. We do copy modulemd defaults in `multi_copy_units`, but nowhere else in the file.
Comment 6 Ian Ballou 2023-09-14 19:01:02 UTC 
Created redmine issue https://projects.theforeman.org/issues/36749 from this bug
Comment 7 Ian Ballou 2023-09-18 21:36:39 UTC 
I wasn't able to reproduce this on the latest code and took a closer look on Satellite 6.13 -- it seems that Pulp isn't syncing modulemd_defaults from the RHEL 9 AppStream repository for some reason.  Not only does the content view repository not have modulemd_defaults, but the library repository is missing them as well.

The filtering matters due to mirroring -- the mirror taken directly from the upstream has the modulemd_default information already in it. However, when telling Pulp to generate the modular metadata itself via either filtering or repository republishing, the modulemd_default information disappears.

RHEL 9 AppStream Library on Jan's reproducer:

```
    "present": {
      "rpm.advisory": {
        "count": 2904,
        "href": "/pulp/api/v3/content/rpm/advisories/?repository_version=/pulp/api/v3/repositories/rpm/rpm/85686487-5c99-4be3-b3a1-a7f956606a62/versions/39/"
      },
      "rpm.modulemd": {
        "count": 14,
        "href": "/pulp/api/v3/content/rpm/modulemds/?repository_version=/pulp/api/v3/repositories/rpm/rpm/85686487-5c99-4be3-b3a1-a7f956606a62/versions/39/"
      },
      "rpm.package": {
        "count": 11796,
        "href": "/pulp/api/v3/content/rpm/packages/?repository_version=/pulp/api/v3/repositories/rpm/rpm/85686487-5c99-4be3-b3a1-a7f956606a62/versions/39/"
      },
      "rpm.packagecategory": {
        "count": 5,
        "href": "/pulp/api/v3/content/rpm/packagecategories/?repository_version=/pulp/api/v3/repositories/rpm/rpm/85686487-5c99-4be3-b3a1-a7f956606a62/versions/39/"
      },
      "rpm.packageenvironment": {
        "count": 4,
        "href": "/pulp/api/v3/content/rpm/packageenvironments/?repository_version=/pulp/api/v3/repositories/rpm/rpm/85686487-5c99-4be3-b3a1-a7f956606a62/versions/39/"
      },
      "rpm.packagegroup": {
        "count": 63,
        "href": "/pulp/api/v3/content/rpm/packagegroups/?repository_version=/pulp/api/v3/repositories/rpm/rpm/85686487-5c99-4be3-b3a1-a7f956606a62/versions/39/"
      },
      "rpm.repo_metadata_file": {
        "count": 1,
        "href": "/pulp/api/v3/content/rpm/repo_metadata_files/?repository_version=/pulp/api/v3/repositories/rpm/rpm/85686487-5c99-4be3-b3a1-a7f956606a62/versions/39/"
      }
    }
```

On my 6.13 reproducer:

```
    "present": {
      "rpm.advisory": {
        "count": 2935,
        "href": "/pulp/api/v3/content/rpm/advisories/?repository_version=/pulp/api/v3/repositories/rpm/rpm/114633d1-a9ce-4df2-94ce-25910d361cc7/versions/2/"
      },
      "rpm.modulemd": {
        "count": 14,
        "href": "/pulp/api/v3/content/rpm/modulemds/?repository_version=/pulp/api/v3/repositories/rpm/rpm/114633d1-a9ce-4df2-94ce-25910d361cc7/versions/2/"
      },
      "rpm.package": {
        "count": 12041,
        "href": "/pulp/api/v3/content/rpm/packages/?repository_version=/pulp/api/v3/repositories/rpm/rpm/114633d1-a9ce-4df2-94ce-25910d361cc7/versions/2/"
      },
      "rpm.packagecategory": {
        "count": 5,
        "href": "/pulp/api/v3/content/rpm/packagecategories/?repository_version=/pulp/api/v3/repositories/rpm/rpm/114633d1-a9ce-4df2-94ce-25910d361cc7/versions/2/"
      },
      "rpm.packageenvironment": {
        "count": 4,
        "href": "/pulp/api/v3/content/rpm/packageenvironments/?repository_version=/pulp/api/v3/repositories/rpm/rpm/114633d1-a9ce-4df2-94ce-25910d361cc7/versions/2/"
      },
      "rpm.packagegroup": {
        "count": 63,
        "href": "/pulp/api/v3/content/rpm/packagegroups/?repository_version=/pulp/api/v3/repositories/rpm/rpm/114633d1-a9ce-4df2-94ce-25910d361cc7/versions/2/"
      },
      "rpm.repo_metadata_file": {
        "count": 1,
        "href": "/pulp/api/v3/content/rpm/repo_metadata_files/?repository_version=/pulp/api/v3/repositories/rpm/rpm/114633d1-a9ce-4df2-94ce-25910d361cc7/versions/2/"
      }
    }

```


I'm changing this into a Pulp BZ since Katello isn't filtering out the modulemd_defaults.

@dalley do you know why  the modulemd_defaults might not be getting synced?
Comment 8 Daniel Alley 2023-09-18 21:39:45 UTC 
>>  I wasn't able to reproduce this on the latest code and took a closer look on Satellite 6.13 -- it seems that Pulp isn't syncing modulemd_defaults from the RHEL 9 AppStream repository for some reason.  Not only does the content view repository not have modulemd_defaults, but the library repository is missing them as well.

RHEL 9 doesn't have any modulemd_defaults (and will never).  They got rid of them after RHEL 8

The "defaults" are just nonmodular packages and if you want some alternate stream then there are modules for those.
Comment 9 Ian Ballou 2023-09-18 21:57:14 UTC 
Daniel, if the modulemd_defaults in the repodata are not needed in RHEL 9, how would you handle this BZ? It seems there is a noticeable difference with the default symbol going away in the DNF report, so users might continue to report it as a bug. 

We might need a KCS to explain why the defaults, useful or not, might go missing.
Comment 10 Daniel Alley 2023-09-18 22:07:04 UTC 
Hold up, I'm gonna need to as some questions because there aren't supposed to be any module defaults in the RHEL 9 appstream repo so far as I know, but there are...
Comment 11 Daniel Alley 2023-09-20 19:09:41 UTC 
Ok, it's just an overlap in meanings.  There are "modulemd-defaults" metadata in modules.yaml, but they don't define any default modules, they just list the profiles.

I'm not able to reproduce this on the latest versions of the code.  If it's broken on older versions, I'd guess that probably libmodulemd is being picky about something in the metadata and silently ignoring them (again).   As of 6.14 we don't use libmodulemd anymore so that might explain why it would be fine now.

I don't have time to dig deeper at the moment
Comment 13 Pavel Moravec 2024-03-25 13:39:36 UTC 
Since the bug is present only in 6.13 (I can confirm 6.14 has it fixed) and it was hit just by one customer that upgraded to 6.14, and since it is low severity bug, I would suggest closing the Jira and BZ. Let me know if somebody insists on fixing this on 6.13.z .

Relevant KCS is already created & linked.
Note You need to log in before you can comment on or make changes to this bug.