Fix is proposed to upstream. See below.

+++ This bug was initially created as a clone of Bug #1844287 +++

With anaconda-33.18-1.fc33 , all our install tests that use a repo accessed via NFS fail. We have four tests. install_repository_nfs_graphical boots the installer normally, then goes to INSTALLATION SOURCE, selects 'nfs' from the "On the network" dropdown, and types in "10.0.2.110:/repo" to the URL box, where that is a working repo. install_repository_nfs_variation passes `inst.repo=nfs:nfsvers=4:10.0.2.110:/repo` on the cmdline, where that is a working package repository. install_repository_nfsiso_variation passes `inst.repo=nfs:nfsvers=4:10.0.2.110:/iso/image.iso` on the cmdline, where that is an installer ISO. All three of those tests fail, showing this message in the logs:

11:13:21,768 ERR payload.manager: PayloadError: Nothing useful found for NFS source at nfs:10.0.2.110:/repo
11:13:21,769 DBG payload.manager: Updating payload thread state: ERROR

We have a couple of other NFS-related tests that pass - install_updates_nfs and install_kickstart_nfs - but they do something slightly different. install_updates_nfs passes `inst.stage2=nfs:nfsvers=4:10.0.2.110:/repo` on the cmdline, so it retrieves the installer environment itself via NFS, but doesn't use it as a repository. `install_kickstart_nfs` similarly retrieves a kickstart via NFS, but that kickstart doesn't use an NFS-accessed repository. So it's specifically using a *repository* accessed via NFS that seems to be the problem.

This seems like a clear violation of Beta criterion "When using the dedicated installer images, the installer must be able to use HTTP, FTP and NFS repositories as package sources", proposing as a Beta blocker.

--- Additional comment from Jiri Konecny on 2020-06-16 16:17:10 UTC ---

Hi Adam,

Could you please add here logs or link to a failing OpenQA test?

--- Additional comment from Adam Williamson on 2020-06-16 17:57:58 UTC ---

Here are logs from a test that used `inst.repo=nfs:nfsvers=4:10.0.2.110:/repo` on the cmdline.

--- Additional comment from Adam Williamson on 2020-06-16 17:58:18 UTC ---



--- Additional comment from Adam Williamson on 2020-06-16 17:58:55 UTC ---



--- Additional comment from Adam Williamson on 2020-06-16 17:59:38 UTC ---



--- Additional comment from Adam Williamson on 2020-06-16 17:59:53 UTC ---



--- Additional comment from Adam Williamson on 2020-06-16 18:00:09 UTC ---



--- Additional comment from Adam Williamson on 2020-06-16 18:01:13 UTC ---



--- Additional comment from Adam Williamson on 2020-06-16 18:01:46 UTC ---



--- Additional comment from Adam Williamson on 2020-06-18 19:51:04 UTC ---

OK, so I figured this out some more. We have two different cases here.

1. The NFS ISO case seems to be really broken. If you specify 'inst.repo=nfs:nfsvers=4:10.0.2.110:/iso/image.iso', the logs indicate that anaconda/dracut try literally to mount '10.0.2.110:/iso/image.iso' (rather than mounting '10.0.2.110:/iso' and then looking for the file 'image.iso'), and obviously that fails because you can't *do* that:

18:01:40,672 WARNING org.fedoraproject.Anaconda.Modules.Payloads:DEBUG:anaconda.modules.payloads.source.nfs.initialization:Setting up NFS source: nfs:nfsvers=4:10.0.2.110:/iso/image.iso
18:01:40,673 WARNING org.fedoraproject.Anaconda.Modules.Payloads:INFO:program:Running... mount -t nfs -o nfsvers=4,nolock 10.0.2.110:/iso/image.iso /run/install/source/mount-0000-nfs-device
18:01:40,779 NOTICE kernel:FS-Cache: Loaded
18:01:40,903 NOTICE kernel:FS-Cache: Netfs 'nfs' registered for caching
18:01:40,927 NOTICE kernel:Key type dns_resolver registered
18:01:41,158 NOTICE kernel:NFS: Registering the id_resolver key type
18:01:41,158 NOTICE kernel:Key type id_resolver registered
18:01:41,158 NOTICE kernel:Key type id_legacy registered
18:01:41,269 INFO kernel:mount.nfs (1899) used greatest stack depth: 12072 bytes left
18:01:41,271 WARNING org.fedoraproject.Anaconda.Modules.Payloads:INFO:program:stderr:
18:01:41,271 WARNING org.fedoraproject.Anaconda.Modules.Payloads:INFO:program:b'mount.nfs: mount spec 10.0.2.110:/iso/image.iso or point /run/install/source/mount-0000-nfs-device is not a directory'

2. All the other cases turn out to be something different and not so clear cut. I figured out it was failing because of the `verify_valid_installtree` check in pyanaconda/payload/image.py . That checks for a `.treeinfo` file. I was setting up the NFS repo on the server by mounting the Server DVD at /mnt/iso and then doing `cp -R /mnt/iso/* /repo` , and exporting /repo via NFS. That command does not copy the `.treeinfo` file, and so the check failed. I changed the command to `rsync -av /mnt/iso/ /repo`, and now it works. So there's definitely an element of me doing something wrong, there. Still, the behaviour changed - it used to work fine *without* the .treeinfo file, now it's required, it seems. It's odd, because the `verify_valid_installtree` check itself has been around since 2018. Perhaps it previously wasn't run, or it was run but the failure wasn't fatal?

I've updated the summary for issue #2 (and will unpropose as a blocker, since this does work in a 'normal' case), I'll file a new bug for issue #1. Thanks!

--- Additional comment from Jiri Konecny on 2020-06-19 09:08:48 UTC ---

We should work even without the .treeinfo file. I'll look onto that.

--- Additional comment from Jiri Konecny on 2020-06-19 14:59:03 UTC ---

PR: https://github.com/rhinstaller/anaconda/pull/2674