Bug 2059373 - Publication creation (during migration to pulp3 as well) can fail if /var/lib/pulp is NFS share
Summary: Publication creation (during migration to pulp3 as well) can fail if /var/lib...
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Pulp
Version: 6.9.8
Hardware: All
OS: All
high vote
Target Milestone: 6.10.4
Assignee: satellite6-bugs
QA Contact: Lai
Depends On:
TreeView+ depends on / blocked
Reported: 2022-02-28 22:14 UTC by Brad Buckingham
Modified: 2022-03-29 14:49 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 2041508
Last Closed: 2022-03-29 14:48:55 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github pulp pulp_rpm issues 2379 0 None closed NFS issue during publish due to not all file descriptors being closed 2022-02-28 22:14:47 UTC
Red Hat Product Errata RHBA-2022:1116 0 None None None 2022-03-29 14:49:09 UTC

Comment 10 Lai 2022-03-22 18:56:14 UTC
Steps to retest:

1. Have a satellite spin up to work on
2. Spin up a rhel box for the NFS share
3. On both systems, run: "rpm -q nfs-utils"
	a. This checks if the system has nfs already installed (it should)
4. On NFS share, run "systemctl status nfs" to check if NFS is running or enabled
5. If not, then enabled the it via "systemctl enable nfs --now" and recheck status.  It should be active.
6. run: “systemctl start nfs-server”, then “systemctl status nfs-server” to ensure that running NFS-server is running.
7. Ensure that the dir /mnt is blank because we're using this location for the export but you can put it anywhere or create a separate dir for this
8. Create an export point by adding "/mnt <sat-server>(rw)" to /etc/exports using vim
9. Stop firewall just in case: systemctl stop firewalld
10. Run: exportfs -r 
11. On the satellite, create a separate folder (pulp1)
12. Grant all permission to pulp1 and /var/lib/pulp/ by running: chown -R pulp:pulp /var/lib/pulp/ and chown -R pulp:pulp pulp1
13. Stop firewall here as well just in case: systemctl stop firewalld
14. run: mount -t nfs <nfs-share server name>:/mnt pulp1 -v
	a. It should output something like this for status:
		mount.nfs: timeout set for Fri Mar 18 14:25:52 2022
		mount.nfs: trying text-based options
15. copy everything in /var/lib/pulp to the share: cp -r /var/lib/pulp/* pulp1
16. On NFS, check /mnt to ensure that the folder structure of /var/lib/pulp is in there (it should have assets, import, exports, tmp, etc)
17. On NFS, create pulp user and grant necessary permissions: adduser pulp, chown -R pulp:pulp /mnt
18. run: umount <NFS share hostname>/mnt
19. Remount the NFS share to /var/lib/pulp/ with: mount -t nfs <nfs-share server name>:/mnt /var/lib/pulp/ -v
        a. Steps 14-18 is necessary so the NFS share has the same file structure as /var/lib/pulp in the export in /mnt.  Then we can mount to /var/lib/pulp/ so that any writing into that dir in satellite can 
                be directed to the NFS share one.  If we don't do this, and we mount directly to /var/lib/pulp/, then pulp cannot work if all its contents was removed by mounting the empty filesystem
20. run: restorecon -Rv /var/lib/pulp
21. Restart pulp services or all services on satellite: foreman-maintain service restart
22. run: showmount -e <NSF share hostname> to ensure mount is still active
23. Create a custom repo and sync
24. Create a cv, add the repos, and publish

Expected result:
step 23 and 24 should complete successfully

Step 23 and 24 does complete successfully

To ensure that NFS share does populate correctly, I checked /mnt/media/artifact and the folder was populated.  Prior, the artifact folder wasn't created because I haven't synced any repos yet.  After syncing, content should appear in "artifact", which it did in my case and you can see that below:

# ls /mnt/media/artifact
01  09  11  18  1d  22  26  31  35  3c  43  49  57  5e  63  6a  73  77  7f  86  92  98  9d  a9  b0  b7  ca  d7  e1  ef  fd
03  0a  15  19  1e  23  2c  32  36  3e  44  4a  5a  60  65  6b  74  78  80  87  94  9a  a0  aa  b3  bc  cd  d8  e6  f4  fe
05  0d  16  1a  1f  24  2d  33  38  41  46  4b  5c  61  68  6c  75  7a  82  8c  95  9b  a6  ad  b5  c0  d1  d9  e8  f6  ff
07  10  17  1c  21  25  30  34  3b  42  48  53  5d  62  69  70  76  7b  83  91  96  9c  a7  ae  b6  c4  d6  dc  ee  f8

Verified on 6.10.4 snap 3 with python3-pulpcore-3.14.14-1.el7pc.noarch

Comment 15 errata-xmlrpc 2022-03-29 14:48:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Satellite 6.10.4 Async Bug Fix Update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.