Description of problem:
When synchronizing Oracle Linux 9 AppStream (after RHEL9 AppStream synced), newly published CVs with filters will have broken modularity metadata.

The reason is shown on an example of ruby 3.1 module stream.

RHEL9 released this snippet for ruby 3.1 module stream:

document: modulemd
version: 2
data:
  name: ruby
  stream: "3.1"
  version: 9010020220623124214
  context: 9
  static_context: true
  arch: x86_64
..
  artifacts:
    rpms:
    - ruby-0:3.1.2-141.module+el9.1.0+15737+76195479.i686
    - ruby-0:3.1.2-141.module+el9.1.0+15737+76195479.src
    - ruby-0:3.1.2-141.module+el9.1.0+15737+76195479.x86_64
    - ruby-bundled-gems-0:3.1.2-141.module+el9.1.0+15737+76195479.i686
..


Few months later on, Oracle Linux 9 AppStream was released, with ruby 3.1 module stream:

document: modulemd
version: 2
data:
  name: ruby
  stream: "3.1"
  version: 9010020220623124214
  context: 9
  static_context: true
  arch: x86_64
..
  artifacts:
    rpms:
    - ruby-0:3.1.2-141.module+el9.1.0+20815+286161bd.i686
    - ruby-0:3.1.2-141.module+el9.1.0+20815+286161bd.x86_64
    - ruby-bundled-gems-0:3.1.2-141.module+el9.1.0+20815+286161bd.i686

Same module (with uniq identification name+stream+version+context+arch, see https://github.com/pulp/pulp_rpm/blob/main/pulp_rpm/app/models/modulemd.py#L73 (_pulp_domain is not applicable). Same module but different content. Somebody is unable to even properly copy from us, sigh.

Now, the problem is pulp stores the modularity data with uniq index:

    "rpm_modulemd_name_stream_version_context_arch_f6598e4e_uniq" UNIQUE CONSTRAINT, btree (name, stream, version, context, arch)

and when syncing a repo, "insert or update" method is used (to allow fixing errors in released modules). So syncing RHEL9 creates this record:

su - postgres -c "psql pulpcore -c \"SELECT * FROM rpm_modulemd WHERE version = '9010020220623124214';\"" | sed 's/ //g' | grep -e artifacts -e "3.1.2-141.module+el9.1.0"

content_ptr_id|name|stream|version|context|arch|dependencies|artifacts|static_context|snippet|description|profiles
92ada8ba-3bdb-4441-b87b-3fcf1999a999|ruby|3.1|9010020220623124214|9|x86_64|[{"platform":["el9"]}]|["ruby-0:3.1.2-141.module+el9.1.0+15737+76195479.i686","ruby-0:3.1.2-141.module+el9.1.0+15737+76195479.src","ruby-0:3.1.2-141.module+el9.1.0+15737+76195479.x86_64","ruby-bundled-gems-0:3.1.2-141.module+el9.1.0+15737+76195479.i686","ruby-bundled-gems-0:3.1.2-141.module+el9.1.0+15737+76195479.x86_64","ruby-bundled-gems-debuginfo-0:3.1.2-141.module+el9.1.0+15737+76195479.i686","ruby-bundled-gems-debuginfo-0:3.1.2-141.module+el9.1.0+15737+76195479.x86_64","ruby-debuginfo-0:3.1.2-141.module+el9.1.0+15737+76195479.i686","ruby-debuginfo-0:3.1.2-141.module+el9.1.0+15737+76195479.x86_64","ruby-debugsource-0:3.1.2-141.module+el9.1.0+15737+76195479.i686","ruby-debugsource-0:3.1.2-141.module+el9.1.0+15737+76195479.x86_64","ruby-default-gems-0:3.1.2-141.module+el9.1.0+15737+76195479.noarch","ruby-devel-0:3.1.2-141.module+el9.1.0+15737+76195479.i686","ruby-devel-0:3.1.2-141.module+el9.1.0+15737+76195479.x86_64","ruby-doc-0:3.1.2-141.module+el9.1.0+15737+76195479.noarch","ruby-libs-0:3.1.2-141.module+el9.1.0+15737+76195479.i686","ruby-libs-0:3.1.2-141.module+el9.1.0+15737+76195479.x86_64","ruby-libs-debuginfo-0:3.1.2-141.module+el9.1.0+15737+76195479.i686","ruby-libs-debuginfo-0:3.1.2-141.module+el9.1.0+15737+76195479.x86_64","rubygem-bigdecimal-0:3.1.1-141.module+el9.1.0+15737+76195479.i686","rubygem-bigdecimal-0:3.1.1-141.module+el9.1.0+15737+76195479.x86_64","rubygem-bigdecimal-debuginfo-0:3.1.1-141.module+el9.1.0+15737+76195479.i686","rubygem-bigdecimal-debuginfo-0:3.1.1-141.module+el9.1.0+15737+76195479.x86_64","rubygem-bundler-0:2.3.7-141.module+el9.1.0+15737+76195479.noarch","rubygem-io-console-0:0.5.11-141.module+el9.1.0+15737+76195479.i686","rubygem-io-console-0:0.5.11-141.module+el9.1.0+15737+76195479.x86_64","rubygem-io-console-debuginfo-0:0.5.11-141.module+el9.1.0+15737+76195479.i686","rubygem-io-console-debuginfo-0:0.5.11-141.module+el9.1.0+15737+76195479.x86_64","rubygem-irb-0:1.4.1-141.module+el9.1.0+15737+76195479.noarch","rubygem-json-0:2.6.1-141.module+el9.1.0+15737+76195479.i686","rubygem-json-0:2.6.1-141.module+el9.1.0+15737+76195479.x86_64","rubygem-json-debuginfo-0:2.6.1-141.module+el9.1.0+15737+76195479.i686","rubygem-json-debuginfo-0:2.6.1-141.module+el9.1.0+15737+76195479.x86_64","rubygem-minitest-0:5.15.0-141.module+el9.1.0+15737+76195479.noarch","rubygem-mysql2-0:0.5.4-1.module+el9.1.0+15737+76195479.src","rubygem-mysql2-0:0.5.4-1.module+el9.1.0+15737+76195479.x86_64","rubygem-mysql2-debuginfo-0:0.5.4-1.module+el9.1.0+15737+76195479.x86_64","rubygem-mysql2-debugsource-0:0.5.4-1.module+el9.1.0+15737+76195479.x86_64","rubygem-mysql2-doc-0:0.5.4-1.module+el9.1.0+15737+76195479.noarch","rubygem-pg-0:1.3.5-1.module+el9.1.0+15737+76195479.src","rubygem-pg-0:1.3.5-1.module+el9.1.0+15737+76195479.x86_64","rubygem-pg-debuginfo-0:1.3.5-1.module+el9.1.0+15737+76195479.x86_64","rubygem-pg-debugsource-0:1.3.5-1.module+el9.1.0+15737+76195479.x86_64","rubygem-pg-doc-0:1.3.5-1.module+el9.1.0+15737+76195479.noarch","rubygem-power_assert-0:2.0.1-141.module+el9.1.0+15737+76195479.noarch","rubygem-psych-0:4.0.3-141.module+el9.1.0+15737+76195479.i686","rubygem-psych-0:4.0.3-141.module+el9.1.0+15737+76195479.x86_64","rubygem-psych-debuginfo-0:4.0.3-141.module+el9.1.0+15737+76195479.i686","rubygem-psych-debuginfo-0:4.0.3-141.module+el9.1.0+15737+76195479.x86_64","rubygem-rake-0:13.0.6-141.module+el9.1.0+15737+76195479.noarch","rubygem-rbs-0:2.1.0-141.module+el9.1.0+15737+76195479.i686","rubygem-rbs-0:2.1.0-141.module+el9.1.0+15737+76195479.x86_64","rubygem-rbs-debuginfo-0:2.1.0-141.module+el9.1.0+15737+76195479.i686","rubygem-rbs-debuginfo-0:2.1.0-141.module+el9.1.0+15737+76195479.x86_64","rubygem-rdoc-0:6.4.0-141.module+el9.1.0+15737+76195479.noarch","rubygem-rexml-0:3.2.5-141.module+el9.1.0+15737+76195479.noarch","rubygem-rss-0:0.2.9-141.module+el9.1.0+15737+76195479.noarch","rubygem-test-unit-0:3.5.3-141.module+el9.1.0+15737+76195479.noarch","rubygem-typeprof-0:0.21.2-141.module+el9.1.0+15737+76195479.noarch","rubygems-0:3.3.7-141.module+el9.1.0+15737+76195479.noarch","rubygems-devel-0:3.3.7-141.module+el9.1.0+15737+76195479.noarch"]|t|---+|Rubyistheinterpretedscriptinglanguageforquickandeasyobject-orientedprogramming.Ithasmanyfeaturestoprocesstextfilesandtodosystemmanagementtasks(asinPerl).Itissimple,straight-forward,andextensible.|{"common":["ruby"]}
|||||||||artifacts:+||
|||||||||-ruby-0:3.1.2-141.module+el9.1.0+15737+76195479.i686+||
|||||||||-ruby-0:3.1.2-141.module+el9.1.0+15737+76195479.src+||
|||||||||-ruby-0:3.1.2-141.module+el9.1.0+15737+76195479.x86_64+||
|||||||||-ruby-bundled-gems-0:3.1.2-141.module+el9.1.0+15737+76195479.i686+||
|||||||||-ruby-bundled-gems-0:3.1.2-141.module+el9.1.0+15737+76195479.x86_64+||
|||||||||-ruby-bundled-gems-debuginfo-0:3.1.2-141.module+el9.1.0+15737+76195479.i686+||
|||||||||-ruby-bundled-gems-debuginfo-0:3.1.2-141.module+el9.1.0+15737+76195479.x86_64+||
|||||||||-ruby-debuginfo-0:3.1.2-141.module+el9.1.0+15737+76195479.i686+||
|||||||||-ruby-debuginfo-0:3.1.2-141.module+el9.1.0+15737+76195479.x86_64+||
|||||||||-ruby-debugsource-0:3.1.2-141.module+el9.1.0+15737+76195479.i686+||
|||||||||-ruby-debugsource-0:3.1.2-141.module+el9.1.0+15737+76195479.x86_64+||
|||||||||-ruby-default-gems-0:3.1.2-141.module+el9.1.0+15737+76195479.noarch+||
|||||||||-ruby-devel-0:3.1.2-141.module+el9.1.0+15737+76195479.i686+||
|||||||||-ruby-devel-0:3.1.2-141.module+el9.1.0+15737+76195479.x86_64+||
|||||||||-ruby-doc-0:3.1.2-141.module+el9.1.0+15737+76195479.noarch+||
|||||||||-ruby-libs-0:3.1.2-141.module+el9.1.0+15737+76195479.i686+||
|||||||||-ruby-libs-0:3.1.2-141.module+el9.1.0+15737+76195479.x86_64+||
|||||||||-ruby-libs-debuginfo-0:3.1.2-141.module+el9.1.0+15737+76195479.i686+||
|||||||||-ruby-libs-debuginfo-0:3.1.2-141.module+el9.1.0+15737+76195479.x86_64+||

but subsequent syncing OL9 causes this:

content_ptr_id|name|stream|version|context|arch|dependencies|artifacts|static_context|snippet|description|profiles
92ada8ba-3bdb-4441-b87b-3fcf1999a999|ruby|3.1|9010020220623124214|9|x86_64|[{"platform":["el9"]}]|["ruby-0:3.1.2-141.module+el9.1.0+15737+76195479.i686","ruby-0:3.1.2-141.module+el9.1.0+15737+76195479.src","ruby-0:3.1.2-141.module+el9.1.0+15737+76195479.x86_64","ruby-bundled-gems-0:3.1.2-141.module+el9.1.0+15737+76195479.i686","ruby-bundled-gems-0:3.1.2-141.module+el9.1.0+15737+76195479.x86_64","ruby-bundled-gems-debuginfo-0:3.1.2-141.module+el9.1.0+15737+76195479.i686","ruby-bundled-gems-debuginfo-0:3.1.2-141.module+el9.1.0+15737+76195479.x86_64","ruby-debuginfo-0:3.1.2-141.module+el9.1.0+15737+76195479.i686","ruby-debuginfo-0:3.1.2-141.module+el9.1.0+15737+76195479.x86_64","ruby-debugsource-0:3.1.2-141.module+el9.1.0+15737+76195479.i686","ruby-debugsource-0:3.1.2-141.module+el9.1.0+15737+76195479.x86_64","ruby-default-gems-0:3.1.2-141.module+el9.1.0+15737+76195479.noarch","ruby-devel-0:3.1.2-141.module+el9.1.0+15737+76195479.i686","ruby-devel-0:3.1.2-141.module+el9.1.0+15737+76195479.x86_64","ruby-doc-0:3.1.2-141.module+el9.1.0+15737+76195479.noarch","ruby-libs-0:3.1.2-141.module+el9.1.0+15737+76195479.i686","ruby-libs-0:3.1.2-141.module+el9.1.0+15737+76195479.x86_64","ruby-libs-debuginfo-0:3.1.2-141.module+el9.1.0+15737+76195479.i686","ruby-libs-debuginfo-0:3.1.2-141.module+el9.1.0+15737+76195479.x86_64","rubygem-bigdecimal-0:3.1.1-141.module+el9.1.0+15737+76195479.i686","rubygem-bigdecimal-0:3.1.1-141.module+el9.1.0+15737+76195479.x86_64","rubygem-bigdecimal-debuginfo-0:3.1.1-141.module+el9.1.0+15737+76195479.i686","rubygem-bigdecimal-debuginfo-0:3.1.1-141.module+el9.1.0+15737+76195479.x86_64","rubygem-bundler-0:2.3.7-141.module+el9.1.0+15737+76195479.noarch","rubygem-io-console-0:0.5.11-141.module+el9.1.0+15737+76195479.i686","rubygem-io-console-0:0.5.11-141.module+el9.1.0+15737+76195479.x86_64","rubygem-io-console-debuginfo-0:0.5.11-141.module+el9.1.0+15737+76195479.i686","rubygem-io-console-debuginfo-0:0.5.11-141.module+el9.1.0+15737+76195479.x86_64","rubygem-irb-0:1.4.1-141.module+el9.1.0+15737+76195479.noarch","rubygem-json-0:2.6.1-141.module+el9.1.0+15737+76195479.i686","rubygem-json-0:2.6.1-141.module+el9.1.0+15737+76195479.x86_64","rubygem-json-debuginfo-0:2.6.1-141.module+el9.1.0+15737+76195479.i686","rubygem-json-debuginfo-0:2.6.1-141.module+el9.1.0+15737+76195479.x86_64","rubygem-minitest-0:5.15.0-141.module+el9.1.0+15737+76195479.noarch","rubygem-mysql2-0:0.5.4-1.module+el9.1.0+15737+76195479.src","rubygem-mysql2-0:0.5.4-1.module+el9.1.0+15737+76195479.x86_64","rubygem-mysql2-debuginfo-0:0.5.4-1.module+el9.1.0+15737+76195479.x86_64","rubygem-mysql2-debugsource-0:0.5.4-1.module+el9.1.0+15737+76195479.x86_64","rubygem-mysql2-doc-0:0.5.4-1.module+el9.1.0+15737+76195479.noarch","rubygem-pg-0:1.3.5-1.module+el9.1.0+15737+76195479.src","rubygem-pg-0:1.3.5-1.module+el9.1.0+15737+76195479.x86_64","rubygem-pg-debuginfo-0:1.3.5-1.module+el9.1.0+15737+76195479.x86_64","rubygem-pg-debugsource-0:1.3.5-1.module+el9.1.0+15737+76195479.x86_64","rubygem-pg-doc-0:1.3.5-1.module+el9.1.0+15737+76195479.noarch","rubygem-power_assert-0:2.0.1-141.module+el9.1.0+15737+76195479.noarch","rubygem-psych-0:4.0.3-141.module+el9.1.0+15737+76195479.i686","rubygem-psych-0:4.0.3-141.module+el9.1.0+15737+76195479.x86_64","rubygem-psych-debuginfo-0:4.0.3-141.module+el9.1.0+15737+76195479.i686","rubygem-psych-debuginfo-0:4.0.3-141.module+el9.1.0+15737+76195479.x86_64","rubygem-rake-0:13.0.6-141.module+el9.1.0+15737+76195479.noarch","rubygem-rbs-0:2.1.0-141.module+el9.1.0+15737+76195479.i686","rubygem-rbs-0:2.1.0-141.module+el9.1.0+15737+76195479.x86_64","rubygem-rbs-debuginfo-0:2.1.0-141.module+el9.1.0+15737+76195479.i686","rubygem-rbs-debuginfo-0:2.1.0-141.module+el9.1.0+15737+76195479.x86_64","rubygem-rdoc-0:6.4.0-141.module+el9.1.0+15737+76195479.noarch","rubygem-rexml-0:3.2.5-141.module+el9.1.0+15737+76195479.noarch","rubygem-rss-0:0.2.9-141.module+el9.1.0+15737+76195479.noarch","rubygem-test-unit-0:3.5.3-141.module+el9.1.0+15737+76195479.noarch","rubygem-typeprof-0:0.21.2-141.module+el9.1.0+15737+76195479.noarch","rubygems-0:3.3.7-141.module+el9.1.0+15737+76195479.noarch","rubygems-devel-0:3.3.7-141.module+el9.1.0+15737+76195479.noarch"]|t|---+|Rubyistheinterpretedscriptinglanguageforquickandeasyobject-orientedprogramming.Ithasmanyfeaturestoprocesstextfilesandtodosystemmanagementtasks(asinPerl).Itissimple,straight-forward,andextensible.|{"common":["ruby"]}
|||||||||artifacts:+||
|||||||||-ruby-0:3.1.2-141.module+el9.1.0+20815+286161bd.i686+||
|||||||||-ruby-0:3.1.2-141.module+el9.1.0+20815+286161bd.x86_64+||
|||||||||-ruby-bundled-gems-0:3.1.2-141.module+el9.1.0+20815+286161bd.i686+||
|||||||||-ruby-bundled-gems-0:3.1.2-141.module+el9.1.0+20815+286161bd.x86_64+||
|||||||||-ruby-default-gems-0:3.1.2-141.module+el9.1.0+20815+286161bd.noarch+||
|||||||||-ruby-devel-0:3.1.2-141.module+el9.1.0+20815+286161bd.i686+||
|||||||||-ruby-devel-0:3.1.2-141.module+el9.1.0+20815+286161bd.x86_64+||
|||||||||-ruby-doc-0:3.1.2-141.module+el9.1.0+20815+286161bd.noarch+||
|||||||||-ruby-libs-0:3.1.2-141.module+el9.1.0+20815+286161bd.i686+||
|||||||||-ruby-libs-0:3.1.2-141.module+el9.1.0+20815+286161bd.x86_64+||

"artifacts" bits remain the same, but "snippet" is updated. Sadly, the "snippet" is the text used in repository metadata.


Now, when using Satellite6 Content View (with RHEL9 AppStream repo) with filters, metadata are generated and use the above cracked snippet. Such that RHEL9 repo ends up with references to ruby-0:3.1.2-141.module+el9.1.0+20815+286161bd.x86_64 (package solely in OL9) in the module.

Then an attempt to install ruby package (or even rubygem-foreman_scap_client that has ruby as dependant) fails:

No available modular metadata for modular package 'ruby-default-gems-3.1.2-141.module+el9.1.0+15737+76195479.noarch', it cannot be installed on the system


More modules are broken the same way, e.g. nginx.


Gladly, there is a workaround in Complete Sync of RHEL9 repo before publishing an affected CV - see linked KCS. But that is far to be ideal approach..


Version-Release number of selected component (if applicable):
Sat 6.13 (but present in all 6.10+ versions)



How reproducible:
100%



Steps to Reproduce:
1. Sync RHEL9 AppStream repo
2. Sync OL 9 AppStream repo (https://yum.oracle.com/repo/OracleLinux/OL9/appstream/developer/x86_64)
3. Publish a CV with the RHEL9 repo and with some filters (e.g. include everything; or exclude just one package/errata)
4. Have a RHEL9 Host associated to the CV and try to install ruby package (or say rubygem-foreman_scap_client, for SCAP).
   - 4a. Try to install with --downloadonly
   - 4b. Try to really install the packages
5. Optionally, inspect modularity metadata generated for individual distribution paths on Satellite. Use modules-artifact.py script:

from pulp_rpm.app.models.repository import RpmDistribution

for distr in RpmDistribution.objects.all():
    if not distr.publication or not distr.publication.published_metadata:
        continue
    modfilter = distr.publication.published_metadata.filter(relative_path__contains='modules.yaml')
    if modfilter.count() > 0:
        print(f"{modfilter.first().contentartifact_set.first().artifact.file.path}   {distr.base_path}")

and run it:

cat modules-artifact.py | sudo -u pulp PULP_SETTINGS='/etc/pulp/settings.py' DJANGO_SETTINGS_MODULE='pulpcore.app.settings' pulpcore-manager shell

In output like:
/var/lib/pulp/media/artifact/e2/121b7d7d1783a68921b79395e35f809fe7a37b54d6a653b399f19eae64f770   RedHat/Library/cv_rhel9_appstream/content/dist/rhel9/9/x86_64/appstream/os

the artifact is (text or gzip) metadata file for modularity for the given distribution path. Inspect any such artifact file you are interested in to see ruby module details.



Actual results:
4a. downloading packages succeeds. There is no problem with packages themselves.
4b. installing fails with error like:

No available modular metadata for modular package 'ruby-default-gems-3.1.2-141.module+el9.1.0+15737+76195479.noarch', it cannot be installed on the system

5. shows ruby modularity as described at the beginning



Expected results:
4b. installation succeeds
5. ruby modularity in pulp artifacts files for metadata of any RHEL9 repo points to RHEL9 packages.



Additional info:
Oracle Linux 9 is the root cause of the problems. That repo broke the assumption about modularity. We should ask the provider of the repo to fix their repo. But:
- Satellite should be vulnerable to errors. And should not alter RHEL9 repo content in CV just b'cos we synced an evil repo
- we must allow the feature to update given module, in case a fix of broken modularity in a repo is needed
- so we must distinguish unique module *not* only on NSVCA, but e.g. on a hash of its content (likewise we do for RPM or errata)