Bug 2061715 - Publication creation (during migration to pulp3 as well) can fail if /var/lib/pulp is NFS share
Summary: Publication creation (during migration to pulp3 as well) can fail if /var/lib...
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Pulp
Version: 6.9.8
Hardware: All
OS: All
high vote
Target Milestone: 6.9.9
Assignee: satellite6-bugs
QA Contact: Lai
Depends On:
TreeView+ depends on / blocked
Reported: 2022-03-08 11:57 UTC by Brad Buckingham
Modified: 2022-04-20 20:35 UTC (History)
9 users (show)

Fixed In Version: pulp_rpm-3.11.4
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 2041508
Last Closed: 2022-04-20 20:34:53 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github pulp pulp_rpm issues 2379 0 None closed NFS issue during publish due to not all file descriptors being closed 2022-03-08 11:57:48 UTC
Red Hat Product Errata RHSA-2022:1478 0 None None None 2022-04-20 20:35:01 UTC

Comment 3 Lai 2022-04-14 15:27:22 UTC
Steps to retest:

1. Have a satellite spin up to work on
2. Spin up a rhel box for the NFS share
3. On both systems, run: "rpm -q nfs-utils"
	a. This checks if the system has nfs already installed (it should)
4. On NFS share, run "systemctl status nfs-server" to check if NFS is running or enabled
5. If not, then enabled the it via "systemctl enable nfs-server --now" and recheck status.  It should be active.
6. run: “systemctl start nfs-server”, then “systemctl status nfs-server” to ensure that running NFS-server is running.
7. Ensure that the dir /mnt is blank because we're using this location for the export but you can put it anywhere or create a separate dir for this
8. Create an export point by adding "/mnt <sat-server>(rw)" to /etc/exports using vim
9. Stop firewall just in case: systemctl stop firewalld
10. Run: exportfs -r
11. set permission to write into /mnt with chmod 777 /mnt
12. On the satellite, create a separate folder (pulp1)
13. Grant all permission to pulp1 and /var/lib/pulp/ by running: chown -R pulp:pulp /var/lib/pulp/ and chown -R pulp:pulp pulp1
14. run: mount -t nfs <nfs-share server name>:/mnt pulp1 -v
	a. It should output something like this for status:
		mount.nfs: timeout set for Fri Mar 18 14:25:52 2022
		mount.nfs: trying text-based options
15. copy everything in /var/lib/pulp to the share: cp -r /var/lib/pulp/* pulp1
16. On NFS, check /mnt to ensure that the folder structure of /var/lib/pulp is in there (it should have assets, import, exports, tmp, etc)
17. Check permission of /var/lib/pulp and it's varying directory-> ls -lZ /var/lib/pulp
18. Check permission on /mnt on the NFS to compare the permissions between the two servers
19. Check the uid an gid of sat for any of the users from steps 17: grep <user> /etc/passwd
20. On NFS, create group, user, and grant necessary permissions: groupadd -g <gid> <user>, adduser -g <user> -u <uid> <user>, chown -R <user:user> /mnt
21. Ensure that the uid and gid matches between NFS share and sat: grep <user> /etc/passwd
22. run: umount <NFS share hostname>:/mnt
23. Remount the NFS share to /var/lib/pulp/ with: mount -t nfs <nfs-share server name>:/mnt /var/lib/pulp/ -v
        a. Steps 14-18 is necessary so the NFS share has the same file structure as /var/lib/pulp in the export in /mnt.  Then we can mount to /var/lib/pulp/ so that any writing into that dir in satellite can 
                be directed to the NFS share one.  If we don't do this, and we mount directly to /var/lib/pulp/, then pulp cannot work if all its contents was removed by mounting the empty filesystem
24. run: restorecon -Rv /var/lib/pulp
25. Restart pulp services or all services on satellite: foreman-maintain service restart
26. run: showmount -e <NSF share hostname> to ensure mount is still active
27. Create a custom repo and sync
28. Create a cv, add the repos, and publish

Expected result:
Syncing should complete successfully.
Publishing cv should complete successfully

Actual result:
Syncing completed successfully
Publishing cv completed successfully

I also tried doing migration and it seems to work fine.

Verified on 6.9.9 snap 2

Comment 4 Lai 2022-04-14 16:43:12 UTC
Forgot to include a step.  On the satellite, run 'setenforce permissive' to allow permission to access the necessary dir.

Comment 8 errata-xmlrpc 2022-04-20 20:34:53 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.9.9 Async Bug Fix Update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.