Bug 2069306

Reference comments on bug 2028377 and bug 2059388 for some of the discussion that led to this bugzilla.

Additional background on why this is function is needed from Red Hat's Westfields Patches project:
In regulated environments, it is desirable to maintain a single disconnected CDN for multiple Red Hat customers across one or more agencies, departments, etc. By maintaining a neutral ACL, controlled access CDN, Red Hat avoids complicated legal and regulatory customer issues involving potential data exfill, need-to-know, etc. The disconnected CDN option also avoids having to duplicate every disconnected Satellite's Organization, Content View, and update schedule. 

Based on the Westfields Patches project costs, this saves customers one quarter to one half of a work week that would be spent on building, transferring, and importing multiple Content View exports, which grow every time a customer creates a disconnected Satellite Content View, or Satellite Organization with its own Content Views. The more complicated the Satellite implementation, the less likely customers are to spend thousands of personnel dollars on the CV export, transfer, CV import tasks. Often regulated environment customers are burning DVD and BluRay discs at a significant personnel time cost.

I concur with GlennS comments.  I too support multiple customers and disconnected satellite servers (air gapped).  The each have different manifests (some have HA/LB others just ELS. others neither) and versions (RHEL6, 7, and/or 8).  Yum format repository exports (like in 6.9.x and earlier) allow a simple 'sync' import on these servers based on manifest enabled repositories.  Please implement this ASAP.

One more feature to consider - Repository metadata.  At present, the hammer repository export command with the --since flag produces incremental repository metadata, and it isn't possible to update the CDN's metadata.  Without these steps, Satellite will only show the most recent, incremental repository metadata. 

This is the current solution:
1. After the hammer repository export commands are finished, rsync each exported repository's content to a different folder i.e. /srv/export/<since-date>/. This make copying the resulting tarball content to the disconnected CDN easier.
2. Use the location of the /srv/export/<since-date>/<path>/repomd.xml path to replace the incremental repository metadata with a full copy from /var/lib/pulp/published/yum/https/Default_Organization/Library/<path-to-target-repository>/repodata/
3. Move any exported custom product content, which is in the "Library/custom" folder into "Library/content/custom." This simplifies the CDN path and makes ACL and Application Load Balancer configuration easier.
3. Create a tarball where the first folder is "content/" in /srv/export

Connecting redmine issue https://projects.theforeman.org/issues/34861 from this bug

 https://projects.theforeman.org/issues/34861 'Proposed-solution-from-katello-point-of-view' example is different that a 6.9.x and earlier export.  Need to insure 6.10.x repo export is comatable with sat6.9.x and earlier ;sync' import (with and without mirror on sync enabled).  Sat6.9x export is of the format (sat6 maint repo):
[root@satellite6-udtf os]# ls -lR *
Packages:
total 8
drwxr-xr-x. 2 foreman foreman   88 May  3 16:58 a
drwxr-xr-x. 2 foreman foreman   38 May  3 16:58 f
drwxr-xr-x. 2 foreman foreman  236 May  3 16:58 p
drwxr-xr-x. 2 foreman foreman 4096 May  3 16:58 r
drwxr-xr-x. 2 foreman foreman 4096 May  3 16:58 s

./Packages/a:
total 15520
-rw-r--r--. 1 foreman foreman 7919944 May  3 16:58 ansible-2.4.0.0-1.el7ae.noarch.rpm
-rw-r--r--. 1 foreman foreman 7969060 May  3 16:58 ansible-2.4.2.0-2.el7.noarch.rpm

./Packages/f:
total 452
-rw-r--r--. 1 foreman foreman 461720 May  3 16:58 fio-3.1-2.el7.x86_64.rpm

./Packages/p:
total 988
-rw-r--r--. 1 foreman foreman  39640 May  3 16:58 python2-jmespath-0.9.0-4.el7ae.noarch.rpm
-rw-r--r--. 1 foreman foreman  70644 May  3 16:58 python-ecdsa-0.11-4.el7.noarch.rpm
-rw-r--r--. 1 foreman foreman 118016 May  3 16:58 python-httplib2-0.9.2-1.el7.noarch.rpm
-rw-r--r--. 1 foreman foreman 273472 May  3 16:58 python-paramiko-2.1.1-2.el7ae.noarch.rpm
-rw-r--r--. 1 foreman foreman 500172 May  3 16:58 python-passlib-1.6.5-2.el7.noarch.rpm

./Packages/r:
...

repodata:
total 188
-rw-r--r--. 1 foreman foreman  12964 May  3 16:58 3fa6d41397de92dc7c668e55321ae94c2aa7182e9db5d6c3c411488efce9a115-primary.xml.gz
-rw-r--r--. 1 foreman foreman 129940 May  3 16:58 9663f898dab554381af021a81ec7447e427de6d012243ff23011f4adb3e6bbe9-filelists.xml.gz
-rw-r--r--. 1 foreman foreman    124 May  3 16:58 a27718cc28ec6d71432e0ef3e6da544b7f9d93f6bb7d0a55aacd592d03144b70-comps.xml
-rw-r--r--. 1 foreman foreman  13458 May  3 16:58 b6e068bd8c9450e03a979e6c75083cad0e2f5e4b6663eafd0872c753e78c3484-other.xml.gz
-rw-r--r--. 1 foreman foreman  19934 May  3 16:58 de12d3d246ff0abb84076aaf66866c6b6fd970d3021badc262af36886ba26f1c-updateinfo.xml.gz
-rw-r--r--. 1 foreman foreman   2166 May  3 16:58 repomd.xml

Upstream bug assigned to paji

Upstream bug assigned to paji

Glenn and John,
Let us know if this PR will work for you. It is in review right now, but check the description 

https://github.com/Katello/katello/pull/10183#issue-1290671021

Partha,

If the disconnected satellite did NOT have 'sync on mirror' enabled for the repository would this produce duplicate rpm's since it's now in a different format (not a..z format)?  
Also normally in sat6.9 and earlier we export ALL of the repositories for RHEL6 and bundle them into a single tar.gz,, then we do the same for RHEL7 and RHEL8.
Here's an example script that bundles all of RHEL7 up and configures it as a single tgz to expand and sync on the disconnected servers (with variable enabled manifest repositories)
that we currently use and is a simple way to export (on 6.9.x server)/import (all 6.x servers) - so to ally this to still function would be what I'm looking for :
#!/bin/sh
SAT6_EXPORTDIR="/var/lib/pulp/katello-export"
REL="RHEL79-`date '+%Y%m%d'`"

hammer repository list|grep 'NCL Software - 7Server' |sort -n|cut -d'|' -f1,2
#12  | NCL Software - 7Server - x86_64

hammer repository list |grep 'Extra Packages for Enterprise Linux 7 - x86_64' |sort -n|cut -d'|' -f1,2
#11  | Extra Packages for Enterprise Linux 7 - x86_64

hammer repository list |grep '7 Server'|sort -n|cut -d'|' -f1,2
#1   | Red Hat Enterprise Linux 7 Server RPMs x86_64 7Server
#2   | Red Hat Enterprise Linux 7 Server - Optional RPMs x86_64 7Server
#3   | Red Hat Enterprise Linux 7 Server - Extras RPMs x86_64
#4   | Red Hat Enterprise Linux 7 Server - Supplementary RPMs x86_64 7Server
#6   | Red Hat Enterprise Linux High Availability for RHEL 7 Server RPMs x86_64 7Server
#7   | Red Hat Software Collections RPMs for Red Hat Enterprise Linux 7 Server x86_6...
#9   | Red Hat Ansible Engine 2 RPMs for Red Hat Enterprise Linux 7 Server x86_64
#36  | Red Hat Enterprise Linux 7 Server Kickstart x86_64 7.9
#513 | Red Hat Satellite Tools 6.9 for RHEL 7 Server RPMs x86_64
#848 | Red Hat Satellite Tools 6.10 for RHEL 7 Server RPMs x86_64
#917 | Red Hat Satellite Maintenance 6 for RHEL 7 Server RPMs x86_64
#918 | Red Hat Satellite 6.10 for RHEL 7 Server RPMs x86_64
#919 | Red Hat Satellite 6.9 for RHEL 7 Server RPMs x86_64

for REPO in 12 11 1 2 3 4 6 7 9 36 513 848 917 918 919; do
  echo "Exporting repository ${REPO} to /var/lib/pulp/katello-export ..."
  hammer repository export --id ${REPO}
done

cd ${SAT6_EXPORTDIR}
test -d ${REL} || mkdir ${REL}
chown foreman:foreman ${REL}

for EXPORT in ????????-????-????-????-???????????? ; do
  echo "Bundling ${EXPORT} "
  cd ${EXPORT}/MYORG/Library
  tar cf - *|(cd   ${SAT6_EXPORTDIR}/${REL}; tar xf -)
  cd ${SAT6_EXPORTDIR}
  mv  ${EXPORT}  ${EXPORT}-${REL}
done

# Update listing files for multi-directories for RedHat repo sync/enable to work.
for LISTING in  `find ${SAT6_EXPORTDIR}/${REL} -type f -name 'listing' | sed -r 's|/[^/]+$||' |sort -u`; do
  echo "Updating file ${LISTING}/listing ..."
  cd ${LISTING}
  /usr/bin/ls -1d *|grep -v listing > listing
done

cd ${SAT6_EXPORTDIR}
echo "Zipping exported repo's: ${SAT6_EXPORTDIR}/${REL}"
echo " to: /net/geoint-2240a/vol/sharedrive/iso/RHEL7_SAT6_`date '+%Y%m%d'`.tgz"
echo " ... this will take hours!"
gtar cfz /net/geoint-2240a/vol/sharedrive/iso/RHEL7_SAT6_`date '+%Y%m%d'`.tgz  ${REL}
chmod 644  /net/geoint-2240a/vol/sharedrive/iso/RHEL7_SAT6_`date '+%Y%m%d'`.tgz

Re: If the disconnected satellite did NOT have 'sync on mirror' enabled for the repository would this produce duplicate rpm's since it's now in a different format (not a..z format)?

It should not. But that being said you are correct that it has be in the a..z format and doesn't right now. I have filed a pulp issue for this  https://github.com/pulp/pulpcore/issues/2933 and will have that looked into by the pulp guys.

I just made a comment on the PR. We're getting closer to the ideal. 

Having the ability to replicate the Red Hat CDN "yum" repository format in disconnected environments will make it much easier for regulated environment customers to adopt and stick with Satellite.

I've added a request for the "--since" date flag for use with incremental repository exports, and the ability to include full repository metadata instead of incremental repository metadata with the repository exports. That full-copy repository metadata is essential when delivering disconnected Satellites to customer. 

Let me know if there are any questions.

I just made a comment on the PR. We're getting closer to the ideal. 

Having the ability to replicate the Red Hat CDN "yum" repository format in disconnected environments will make it much easier for regulated environment customers to adopt and stick with Satellite.

I've added a request for the "--since" date flag for use with incremental repository exports, and the ability to include full repository metadata instead of incremental repository metadata with the repository exports. That full-copy repository metadata is essential when delivering disconnected Satellites to customer. 

Let me know if there are any questions.

Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/34861 has been resolved.

Moving it to post now that https://github.com/pulp/pulpcore/pull/2951 is merged.

Glenn, John,
So one issue I noticed while syncing this exported content was issues with https ca cert.
* I exported RHEL 7 Server
* Moved the exacted content to /var/www/html/pub/repos on of the satellites
* Then on the importing sat Subscriptions -> Manage Manifest -> Red Hat Cdn -> updated the CDN url to `https://<fqdn>/pub/repos`
* Tried to enable RHEL 7 (was not able to)

When communicating with RH CDN, only RH's ca_cert is issued as the only trusted cert in the request. This means if the upstream server sends a custom ca we may end up getting SSL Error.

I tried enabling repos with the CDN URL set to 'http://<other-fqdn>/repo'  and it worked ok.

I have filed this as bz here => https://bugzilla.redhat.com/show_bug.cgi?id=2112098

Question is if that is a deal breaker. Should we hold off on this until 2112098 is in?

Partha, I read https://bugzilla.redhat.com/show_bug.cgi?id=2112098, and not sure of the error your seeing.  In my case our 'disconnected' satellites are air gapped.
In the past we had to expand the export (RHEL7 Server) so the Packages and repo directories were available at /var/www/html/pub/repos/content/dist/rhel/server/7/7Server/x86_64/os, then the RH Product would enable and can be imported/synced.  So I don't believe we would hit this - would we?  BTW if there's electronic connections between the sat servers in your example - shouldn't they be satellite/capsule configuration?

Question: Is the CA certificate the Root CA used by the disconnected Satellite to communicate with the Apache CDN server? If so, then that's an easy fix - the disconnected Satellite server must have that Root CA certificate installed to the server and the update-ca-trust export command must be run before the disconnected Satellite can pull content from Apache CDN server.

By using 'hammer content-export ... --format "syncable"' it is possible to create complete export of Library, ContentView version or single repository. Incremental exports in syncable format are not available.

The exported content can be published via http, synced and consumed by hosts.

VERIFIED with Satellite 6.12 SNAP10 @ RHEL8.6

My apologies for taking so long. I was sent to an engagement in Florida, and then California. I finally downloaded the Satellite 6.12 beta repositories and built a test Satellite server. 

One thing is missing from this solution: syncable format incremental exports.  Given the size of RHEL 8 Base OS and AppStream is several hundred Gigabytes, this would require the user or the assigned Red Hat consultant, to transfer all content within the target repository, library, or content view.  Right now, I'm only seeing the --format option for hammer content-export complete options (library, repository, and version) but not for the incremental options. 

Could this be added before Satellite 6.12 goes GA? I'd hate to tell disconnected customers who are supporting multiple customers (including Red Hat) that they have to keep running a Satellite 6.9 server for another year and wait for Satellite 6.13.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.12 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:8506