Bug 727522

Summary: unable to install using nfs repo by askmethod
Product: [Fedora] Fedora Reporter: He Rui <rhe>
Component: anacondaAssignee: Martin Sivák <msivak>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rawhideCC: awilliam, cra, jbwillia, jonathan, robatino, tflink, twu, vanmeeuwen+fedora
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: AcceptedBlocker
Fixed In Version: anaconda-16.15-1 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-10-03 12:04:58 EDT Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Bug Depends On:    
Bug Blocks: 713564    
Attachments:
Description Flags
anaconda.log
none
syslog
none
program.log
none
error log of anaconda 16.19
none
program log
none
syslog on f16 rc2
none
F16-Beta.RC2 anaconda log
none
F16-Beta.RC2 program log
none
F16-Beta.RC2 syslog
none
F16-Beta.RC2 dmesg output capture
none
anaconda logs none

Description He Rui 2011-08-02 07:31:28 EDT
Description of problem:
Boot installer with kernel option askmethod. After inputting a correct nfs server and path, it said 'The url provided doesn't contain an installable tree'.

Version-Release number of selected component (if applicable):
anaconda 16.14
16 alpha tc1

How reproducible:
100%

Steps to Reproduce:
1. boot 16 alpha tc1 x86_64 boot.iso with askmethod 
2. input a correct nfs server and path, click ok
Comment 1 He Rui 2011-08-02 07:34:55 EDT
Created attachment 516301 [details]
anaconda.log
Comment 2 Chris Lumens 2011-08-02 08:54:24 EDT
Can you grab /tmp/program.log and /tmp/syslog as well?  Thanks.
Comment 3 He Rui 2011-08-02 22:54:54 EDT
Created attachment 516417 [details]
syslog
Comment 4 He Rui 2011-08-02 22:55:18 EDT
Created attachment 516418 [details]
program.log
Comment 5 He Rui 2011-08-02 22:58:12 EDT
Oh, it seems the locking problem is the root cause:

02:48:39,486 ERR program: mount.nfs: rpc.statd is not running but is required for remote locking.
mount.nfs: Either use '-o nolock' to keep locks local, or start statd.
Comment 6 Chris Lumens 2011-08-03 10:15:43 EDT
Well, we're passing nolock.  We're just doing it poorly:

02:48:38,659 INFO program: Running... /bin/mount -n -t nfs -o ,nolock nfs.englab.nay.redhat.com:/pub/fedora/linux/releases/15/Fedora/x86_64/os/ /mnt/install/testmnt

Let's play last person to touch it fixes it.
Comment 7 He Rui 2011-08-08 07:03:01 EDT
Proposed as beta blocker.
Comment 8 Tao Wu 2011-08-16 05:35:43 EDT
It still happens on f16 RC4.
Comment 9 David Cantrell 2011-08-17 10:34:00 EDT
Fixed in ef5499e7ab580ffd97de3b26a0e028c37260abba
Comment 10 Tim Flink 2011-09-01 13:16:07 EDT
Discussed in the 2011-08-26 blocker review meeting. Accepted as a Fedora 16 beta blocker as it violates the following beta release criterion [1]:

The installer must be able to use the HTTP, FTP and NFS remote package source options.

[1] https://fedoraproject.org/wiki/Fedora_16_Beta_Release_Criteria
Comment 11 Adam Williamson 2011-09-07 17:20:48 EDT
Tao Wu filed 'pass' results for all NFS install methods in F16 Beta TC1 bar NFS-ISO, so it seems like this is mostly fixed - though he did say he saw this error when trying to install with the NFS ISO method. Anaconda folks, should we close this, or is there still something to fix for the NFS ISO case? See https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=735027 for the NFS ISO fail (he originally had a pilot error using the wrong .iso, but after fixing that, claimed to have seen this error).
Comment 12 Tao Wu 2011-09-13 03:48:12 EDT
According to the test results [1] from boblfoot and mine, I believe this nfs-iso problem has been solved in the F16 Beta TC2 ( anaconda 16.17 ).

[1] https://fedoraproject.org/wiki/Test_Results:Fedora_16_Beta_TC2_Install#General_Tests
Comment 13 Adam Williamson 2011-09-13 21:44:57 EDT
let's CLOSE this, then.
Comment 14 Tao Wu 2011-09-26 04:44:58 EDT
It seems happen again on anaconda 16.19, I will upload the related log files.
Comment 15 Tao Wu 2011-09-26 04:46:10 EDT
Created attachment 524859 [details]
error log of anaconda 16.19
Comment 16 Tao Wu 2011-09-26 04:46:56 EDT
Created attachment 524860 [details]
program log
Comment 17 Tao Wu 2011-09-26 04:47:39 EDT
Created attachment 524861 [details]
syslog on f16 rc2
Comment 18 Adam Williamson 2011-09-26 14:39:51 EDT
So, comment #14 is low on details, but from the results matrix we get a bit more detail. Tao marked the NFS_default and NFSISO_default test cases as failing, but the NFS_variation and NFSISO_variation test cases as passing. The difference between default and variation is that default uses 'askmethod' and then picks NFS from the list, while variation uses the 'repo=' parameter to pass anaconda a repository location.

NFS uses a tree, while ISO uses an ISO located on an NFS server. Apparently both of these are failing.

Bob Lightfoot marked NFSISO_variation as passing (in conflict to twu's report).

We could probably do with a bit more testing here.
Comment 19 Adam Williamson 2011-09-26 14:42:18 EDT
twu's program.log contains:

    /* mount the first image and check for a .treeinfo file */
    checked_asprintf(&buf, "/mnt/install/isodir/%s", files[0]);
    if (doPwMount(buf, "/mnt/install/testmnt", "auto", "ro", NULL)) {
        logMessage(ERROR, "ISO image %s does not contain a .treeinfo file", files[0]);
        goto cleanup3;
    }

So it looks like it's barfing on the .treeinfo file.
Comment 20 Adam Williamson 2011-09-26 15:54:11 EDT
that above message was garbled. twu's program.log shows:

08:38:28,269 INFO program: Running... /bin/mount -n -t auto -o ro /mnt/install/isodir/(null) /mnt/install/testmnt
08:38:28,274 ERR program: mount: special device /mnt/install/isodir/(null) does not exist

and if you look at nfsinstall.c it looks like that failed mount comes from that code.

I looked at a public F16 mirror, the RH internal NFS server that twu used for his install, and the Beta RC2 DVD, and they all have .treeinfo files. So I'm not sure how anaconda wound up with (null).
Comment 21 Tim Flink 2011-09-26 17:32:53 EDT
I'm having trouble reproducing this locally.

I'm using beta RC2 netinstall media, askmethod and pointing it towards my local F16 mirror (i386, synced this morning).

I'm hitting a crash, but it has to do with anaconda and the existing mdadm raid on my test system and IIUC, is after any nfs source issues would have shown up. As far as I can tell, install from NFS is working though.
Comment 22 Adam Williamson 2011-09-26 17:56:32 EDT
I also had trouble reproducing this. I managed to get an nfsiso install to the partitioning stage (where I gave up for other, unrelated reasons, to do with my test system).

Note that in F16 the default nfs client mount behaviour is to use NFSv4, which presents problems. I had to configure my NFS server according to https://wiki.archlinux.org/index.php/NFSv4 in order for my test system to be able to successfully mount it. But once I did that, it worked.

I created a /home/adamw/Downloads/temp directory on my server (my laptop, actually...) and put the Beta RC2 DVD in it. I edited its /etc/exports to read:

/home/adamw/Downloads/temp *(ro,sync,fsid=0,no_subtree_check)

(this is insecure, btw, it puts no restrictions on what addresses can access the share). the fsid=0 part seems to be key for NFSv4. Then I wrote Tim's http://tflink.fedorapeople.org/iso/20110926_libreport.x64.boot.iso to a live USB stick with livecd-iso-to-disk. I booted from the USB stick with 'askmethod', and told it to use NFS. I entered my laptop's IP address as the server and simply / as the path (because with NFSv4 you don't enter the full path to the share, you enter the path relative to the NFS root, which is the share with fsid=0 in its options).

anaconda successfully mounted the share, extracted the ISO, got the .treeinfo file, and proceeded through.

I will re-test with the Beta RC2 boot.iso rather than Tim's, and set things up so I can complete the installation this time, just to confirm. But I expect it will succeed.
Comment 23 Adam Williamson 2011-09-26 18:29:37 EDT
Confirmed the same results with RC2 boot.iso.

I can reproduce the error messages attached in the above comments if I direct anaconda at an NFS directory which is valid on the server but is not the right directory - i.e. does not contain a usable DVD ISO and is not the root of an expanded install tree. The correct directory to point at, btw, is /development/16/{basearch}/os - that's the dir which has the .treeinfo file in it.

tflink tested the 'expanded tree' case and I tested the ISO case, both with askmethod, and it worked for both of us when we used the correct directory. So I'm pretty sure the report is a false alarm.

Let's assume that this bug is pilot error for now, and consider it not to block RC3. Bob, Tao, if you still have trouble with this in further testing, please provide more precise details of the exact config you used to test it, and verify that the NFS server path you're passing to anaconda is actually the right one. You can switch to anaconda's console - ctrl-alt-f2 - to poke around; 'mount' should show you where anaconda has mounted the NFS share, and you can 'ls' it to verify exactly what directory it is.
Comment 24 Robert Lightfoot 2011-09-26 20:40:35 EDT
Created attachment 525016 [details]
F16-Beta.RC2 anaconda log
Comment 25 Robert Lightfoot 2011-09-26 20:41:09 EDT
Created attachment 525017 [details]
F16-Beta.RC2 program log
Comment 26 Robert Lightfoot 2011-09-26 20:41:45 EDT
Created attachment 525018 [details]
F16-Beta.RC2 syslog
Comment 27 Robert Lightfoot 2011-09-26 20:42:54 EDT
Created attachment 525019 [details]
F16-Beta.RC2 dmesg output capture
Comment 28 Adam Williamson 2011-09-26 20:47:42 EDT
Bob's clearly hitting an NFS config issue:

00:34:28,735 ERR program: mount.nfs: access denied by server while mounting 192.168.2.200:/var/www/html/test386/
Comment 29 Hongqing Yang 2011-09-27 07:07:15 EDT
Created attachment 525088 [details]
anaconda logs

reproduced with f16 beta rc3
Comment 30 Adam Williamson 2011-09-27 13:06:23 EDT
that's not 'reproduced', it's a different problem. It's worth looking at anaconda.log to see what's going on.

10:58:18,393 INFO loader: url is nfs:10.66.13.140:/var/ftp/pub/Fedora-16-Beta-x86_64-DVD.iso/.treeinfo
10:58:18,393 DEBUG loader: parseNfsHostPathOpts url: |10.66.13.140:/var/ftp/pub/Fedora-16-Beta-x86_64-DVD.iso/.treeinfo|
10:58:18,393 DEBUG loader: parseNfsHostPathOpts host: |10.66.13.140|
10:58:18,393 DEBUG loader: parseNfsHostPathOpts path: |/var/ftp/pub/Fedora-16-Beta-x86_64-DVD.iso/.treeinfo|
10:58:18,393 DEBUG loader: parseNfsHostPathOpts opts: ||
10:58:18,393 INFO loader: file location: nfs:10.66.13.140:/var/ftp/pub/Fedora-16-Beta-x86_64-DVD.iso/.treeinfo
10:58:18,707 ERR loader: failed to mount nfs source
10:58:18,710 ERR loader: could not unmount /mnt/install/testmnt in getFileFromNfs: Invalid argument
10:58:18,710 DEBUG loader: parseNfsHostPathOpts url: |10.66.13.140:/var/ftp/pub/Fedora-16-Beta-x86_64-DVD.iso|
10:58:18,710 DEBUG loader: parseNfsHostPathOpts host: |10.66.13.140|
10:58:18,710 DEBUG loader: parseNfsHostPathOpts path: |/var/ftp/pub/Fedora-16-Beta-x86_64-DVD.iso|
10:58:18,710 DEBUG loader: parseNfsHostPathOpts opts: ||
10:58:18,780 ERR loader: couldn't mount 10.66.13.140:/var/ftp/pub/Fedora-16-Beta-x86_64-DVD.iso to look for NFSISO
11:00:13,002 INFO loader: url is nfs:10.66.13.140:/var/ftp/pub/Fedora-16-Beta-x86_64-DVD.iso/.treeinfo
11:00:13,002 DEBUG loader: parseNfsHostPathOpts url: |10.66.13.140:/var/ftp/pub/Fedora-16-Beta-x86_64-DVD.iso/.treeinfo|
11:00:13,002 DEBUG loader: parseNfsHostPathOpts host: |10.66.13.140|
11:00:13,002 DEBUG loader: parseNfsHostPathOpts path: |/var/ftp/pub/Fedora-16-Beta-x86_64-DVD.iso/.treeinfo|
11:00:13,002 DEBUG loader: parseNfsHostPathOpts opts: ||
11:00:13,003 INFO loader: file location: nfs:10.66.13.140:/var/ftp/pub/Fedora-16-Beta-x86_64-DVD.iso/.treeinfo
11:00:13,057 ERR loader: failed to mount nfs source
11:00:13,057 ERR loader: could not unmount /mnt/install/testmnt in getFileFromNfs: Invalid argument
11:00:13,057 DEBUG loader: parseNfsHostPathOpts url: |10.66.13.140:/var/ftp/pub/Fedora-16-Beta-x86_64-DVD.iso|
11:00:13,058 DEBUG loader: parseNfsHostPathOpts host: |10.66.13.140|
11:00:13,058 DEBUG loader: parseNfsHostPathOpts path: |/var/ftp/pub/Fedora-16-Beta-x86_64-DVD.iso|
11:00:13,058 DEBUG loader: parseNfsHostPathOpts opts: ||
11:00:13,128 ERR loader: couldn't mount 10.66.13.140:/var/ftp/pub/Fedora-16-Beta-x86_64-DVD.iso to look for NFSISO
11:03:08,653 INFO loader: url is nfs:10.66.13.140:/var/ftp/pub/Fedora-16-Beta-x86_64-DVD.iso/.treeinfo
11:03:08,653 DEBUG loader: parseNfsHostPathOpts url: |10.66.13.140:/var/ftp/pub/Fedora-16-Beta-x86_64-DVD.iso/.treeinfo|
11:03:08,653 DEBUG loader: parseNfsHostPathOpts host: |10.66.13.140|
11:03:08,653 DEBUG loader: parseNfsHostPathOpts path: |/var/ftp/pub/Fedora-16-Beta-x86_64-DVD.iso/.treeinfo|
11:03:08,653 DEBUG loader: parseNfsHostPathOpts opts: ||
11:03:08,653 INFO loader: file location: nfs:10.66.13.140:/var/ftp/pub/Fedora-16-Beta-x86_64-DVD.iso/.treeinfo
11:03:08,734 ERR loader: failed to mount nfs source
11:03:08,734 ERR loader: could not unmount /mnt/install/testmnt in getFileFromNfs: Invalid argument
11:03:08,735 DEBUG loader: parseNfsHostPathOpts url: |10.66.13.140:/var/ftp/pub/Fedora-16-Beta-x86_64-DVD.iso|
11:03:08,735 DEBUG loader: parseNfsHostPathOpts host: |10.66.13.140|
11:03:08,735 DEBUG loader: parseNfsHostPathOpts path: |/var/ftp/pub/Fedora-16-Beta-x86_64-DVD.iso|
11:03:08,735 DEBUG loader: parseNfsHostPathOpts opts: ||
11:03:08,812 ERR loader: couldn't mount 10.66.13.140:/var/ftp/pub/Fedora-16-Beta-x86_64-DVD.iso to look for NFSISO

You entered 10.66.13.140 as the server and /var/ftp/pub/Fedora-16-Beta-x86_64-DVD.iso as the path. That's not the right way to do it: you enter the directory where the ISO lives as the path, not the ISO itself. As you can see from the log, anaconda treated the ISO name as a *directory* and attempted to mount it.

You should have entered /var/ftp/pub as the path.

Please, before deciding you've reproduced this bug, examine the logs and see if they indicate that you've done something wrong with the paths. When in doubt, try mounting the same path anaconda is trying to mount manually and see how that turns out.

All anaconda really does here is parse the path you enter and run 'mount.nfs' on it.
Comment 31 Adam Williamson 2011-09-28 21:56:17 EDT
setting back to VERIFIED, as tflink, myself and southern_gentleman all confirmed success with correct NFS server setup and paths.
Comment 32 Adam Williamson 2011-09-29 05:04:40 EDT
tested again with askmethod / nfsiso in rc4 and it still works, but twu still has trouble, and we're not sure why.

i checked that the nfs server is mounted as /mnt/install/isodir and the iso is then mounted as /mnt/install/source . i checked i had no errors in any logs. and i checked it actually pulled the packages from the NFS server by the simple expedient of dropping the server off the network halfway through package install for 3 minutes, then putting it back; package install hung while it was off the network, then picked up again when I put it back.

so this really, really, really works for me.
Comment 33 Tao Wu 2011-09-29 06:31:32 EDT
I believed it works fine without error at first, but when it proceed through the installer until the 'Software Selection' step,  I saw the repo was not really pointing to the NFS direction if I click "Modify repository".  So it is not clear that what is the real source of repo, DVD or NFS? 

But according to results of Adam's experiment -- bring down the NFS server halfway through install of packages, it proves that the source is the NFS.

So it may not be a problem any more.
Comment 34 Adam Williamson 2011-09-29 07:08:57 EDT
note i actually tested with the netinst image, not the dvd, so you might want to repeat my test with the dvd.
Comment 35 Ben Williams 2011-09-29 10:00:57 EDT
i did 4 installs using the rc3 yesterday with no issues 
useing linux askmethod and pointing to my NFS server and in useing linux repo=nfs:serverip:path and repo=nfsiso:serverip:path

so i see this issue as closed
Comment 36 Adam Williamson 2011-10-03 12:04:58 EDT
yeah, I really think this is good and we can close it; all the issues I've seen appear to be pilot error of some kind.