Bug 1844287

Summary: NFS repos only work if .treeinfo file is present with anaconda-33.18-1.fc33+
Product: [Fedora] Fedora Reporter: Adam Williamson <awilliam>
Component: anacondaAssignee: Jiri Konecny <jkonecny>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: low Docs Contact:
Priority: unspecified    
Version: 33CC: anaconda-maint-list, jkonecny, jonathan, kellin, robatino, vanmeeuwen+fedora, vponcova, wwoods
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard: openqa
Fixed In Version: anaconda-33.20-1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1849093 (view as bug list) Environment:
Last Closed: 2021-04-15 11:22:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
anaconda.log
none
dbus.log
none
dnf.librepo.log
none
packaging.log
none
program.log
none
storage.log
none
syslog none

Description Adam Williamson 2020-06-05 01:29:23 UTC
With anaconda-33.18-1.fc33 , all our install tests that use a repo accessed via NFS fail. We have four tests. install_repository_nfs_graphical boots the installer normally, then goes to INSTALLATION SOURCE, selects 'nfs' from the "On the network" dropdown, and types in "10.0.2.110:/repo" to the URL box, where that is a working repo. install_repository_nfs_variation passes `inst.repo=nfs:nfsvers=4:10.0.2.110:/repo` on the cmdline, where that is a working package repository. install_repository_nfsiso_variation passes `inst.repo=nfs:nfsvers=4:10.0.2.110:/iso/image.iso` on the cmdline, where that is an installer ISO. All three of those tests fail, showing this message in the logs:

11:13:21,768 ERR payload.manager: PayloadError: Nothing useful found for NFS source at nfs:10.0.2.110:/repo
11:13:21,769 DBG payload.manager: Updating payload thread state: ERROR

We have a couple of other NFS-related tests that pass - install_updates_nfs and install_kickstart_nfs - but they do something slightly different. install_updates_nfs passes `inst.stage2=nfs:nfsvers=4:10.0.2.110:/repo` on the cmdline, so it retrieves the installer environment itself via NFS, but doesn't use it as a repository. `install_kickstart_nfs` similarly retrieves a kickstart via NFS, but that kickstart doesn't use an NFS-accessed repository. So it's specifically using a *repository* accessed via NFS that seems to be the problem.

This seems like a clear violation of Beta criterion "When using the dedicated installer images, the installer must be able to use HTTP, FTP and NFS repositories as package sources", proposing as a Beta blocker.

Comment 1 Jiri Konecny 2020-06-16 16:17:10 UTC
Hi Adam,

Could you please add here logs or link to a failing OpenQA test?

Comment 2 Adam Williamson 2020-06-16 17:57:58 UTC
Here are logs from a test that used `inst.repo=nfs:nfsvers=4:10.0.2.110:/repo` on the cmdline.

Comment 3 Adam Williamson 2020-06-16 17:58:18 UTC
Created attachment 1697674 [details]
anaconda.log

Comment 4 Adam Williamson 2020-06-16 17:58:55 UTC
Created attachment 1697675 [details]
dbus.log

Comment 5 Adam Williamson 2020-06-16 17:59:38 UTC
Created attachment 1697676 [details]
dnf.librepo.log

Comment 6 Adam Williamson 2020-06-16 17:59:53 UTC
Created attachment 1697677 [details]
packaging.log

Comment 7 Adam Williamson 2020-06-16 18:00:09 UTC
Created attachment 1697678 [details]
program.log

Comment 8 Adam Williamson 2020-06-16 18:01:13 UTC
Created attachment 1697679 [details]
storage.log

Comment 9 Adam Williamson 2020-06-16 18:01:46 UTC
Created attachment 1697680 [details]
syslog

Comment 10 Adam Williamson 2020-06-18 19:51:04 UTC
OK, so I figured this out some more. We have two different cases here.

1. The NFS ISO case seems to be really broken. If you specify 'inst.repo=nfs:nfsvers=4:10.0.2.110:/iso/image.iso', the logs indicate that anaconda/dracut try literally to mount '10.0.2.110:/iso/image.iso' (rather than mounting '10.0.2.110:/iso' and then looking for the file 'image.iso'), and obviously that fails because you can't *do* that:

18:01:40,672 WARNING org.fedoraproject.Anaconda.Modules.Payloads:DEBUG:anaconda.modules.payloads.source.nfs.initialization:Setting up NFS source: nfs:nfsvers=4:10.0.2.110:/iso/image.iso
18:01:40,673 WARNING org.fedoraproject.Anaconda.Modules.Payloads:INFO:program:Running... mount -t nfs -o nfsvers=4,nolock 10.0.2.110:/iso/image.iso /run/install/source/mount-0000-nfs-device
18:01:40,779 NOTICE kernel:FS-Cache: Loaded
18:01:40,903 NOTICE kernel:FS-Cache: Netfs 'nfs' registered for caching
18:01:40,927 NOTICE kernel:Key type dns_resolver registered
18:01:41,158 NOTICE kernel:NFS: Registering the id_resolver key type
18:01:41,158 NOTICE kernel:Key type id_resolver registered
18:01:41,158 NOTICE kernel:Key type id_legacy registered
18:01:41,269 INFO kernel:mount.nfs (1899) used greatest stack depth: 12072 bytes left
18:01:41,271 WARNING org.fedoraproject.Anaconda.Modules.Payloads:INFO:program:stderr:
18:01:41,271 WARNING org.fedoraproject.Anaconda.Modules.Payloads:INFO:program:b'mount.nfs: mount spec 10.0.2.110:/iso/image.iso or point /run/install/source/mount-0000-nfs-device is not a directory'

2. All the other cases turn out to be something different and not so clear cut. I figured out it was failing because of the `verify_valid_installtree` check in pyanaconda/payload/image.py . That checks for a `.treeinfo` file. I was setting up the NFS repo on the server by mounting the Server DVD at /mnt/iso and then doing `cp -R /mnt/iso/* /repo` , and exporting /repo via NFS. That command does not copy the `.treeinfo` file, and so the check failed. I changed the command to `rsync -av /mnt/iso/ /repo`, and now it works. So there's definitely an element of me doing something wrong, there. Still, the behaviour changed - it used to work fine *without* the .treeinfo file, now it's required, it seems. It's odd, because the `verify_valid_installtree` check itself has been around since 2018. Perhaps it previously wasn't run, or it was run but the failure wasn't fatal?

I've updated the summary for issue #2 (and will unpropose as a blocker, since this does work in a 'normal' case), I'll file a new bug for issue #1. Thanks!

Comment 11 Jiri Konecny 2020-06-19 09:08:48 UTC
We should work even without the .treeinfo file. I'll look onto that.

Comment 12 Jiri Konecny 2020-06-19 14:59:03 UTC
PR: https://github.com/rhinstaller/anaconda/pull/2674

Comment 13 Ben Cotton 2020-08-11 15:31:00 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 33 development cycle.
Changing version to 33.