Bug 879187

Summary: NFS in RC1 regressed vs. TC9: does not override inst.stage2 when passed as inst.repo, interactive NFS is impossible
Product: [Fedora] Fedora Reporter: Adam Williamson <awilliam>
Component: anacondaAssignee: Brian Lane <bcl>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 18CC: BobLfoot, g.kaviyarasu, jonathan, jreznik, kparal, pf.rhlists, robatino, tflink, vanmeeuwen+fedora
Target Milestone: ---Keywords: CommonBugs
Target Release: ---   
Hardware: x86_64   
OS: Unspecified   
Whiteboard: abrt_hash:2bfbc4b2e44b4fcd656616204bf63dfb9002eb985244a985645b33667bc3c242 https://fedoraproject.org/wiki/Common_F18_bugs#nfs-install-broken AcceptedBlocker
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-01-02 21:48:17 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 752661, 876223    
Attachments:
Description Flags
File: anaconda-tb
none
File: product
none
File: type
none
File: ifcfg.log
none
File: storage.log
none
File: version
none
File: environ
none
File: executable
none
File: anaconda.log
none
File: syslog
none
File: hashmarkername
none
File: packaging.log
none
File: cmdline_file
none
File: release
none
File: program.log
none
packaging.log: appended repo=nfsiso:192.168.1.11:/export/fedora (result: Closest Mirror)
none
program.log: appended repo=nfsiso:192.168.1.11:/export/fedora (result: Closest Mirror)
none
syslog: appended repo=nfsiso:192.168.1.11:/export/fedora (result: Closest Mirror)
none
anaconda.log: appended repo=nfsiso:192.168.1.11:/export/fedora (result: Closest Mirror)
none
packaging.log: booted netinst as usual, interactively specified NFS repo (result: Error setting up software source)
none
program.log: booted netinst as usual, interactively specified NFS repo (result: Error setting up software source)
none
anaconda.log: booted netinst as usual, interactively specified NFS repo (result: Error setting up software source)
none
syslog: booted netinst as usual, interactively specified NFS repo (result: Error setting up software source)
none
IRC log extract from when we discussed applying this patch (for forensic purposes)
none
18.36 anaconda-tb for graphical nfsiso
none
18.36 anaconda-tb for cmdline nfsiso none

Description Adam Williamson 2012-11-22 10:05:19 UTC
Description of problem:
Attempting to install from an NFS repository.

Version-Release number of selected component:
anaconda-18.29.2

Additional info:
libreport version: 2.0.17
cmdline:        /usr/bin/python  /sbin/anaconda
kernel:         3.6.6-3.fc18.x86_64

description:
:The following was filed automatically by anaconda:
:anaconda 18.29.2 exception report
:Traceback (most recent call first):
:  File "/usr/lib64/python2.7/site-packages/pyanaconda/image.py", line 41, in findFirstIsoImage
:    flush = os.stat(path)
:  File "/usr/lib64/python2.7/site-packages/pyanaconda/packaging/yumpayload.py", line 620, in _configureBaseRepo
:    image = findFirstIsoImage(path)
:  File "/usr/lib64/python2.7/site-packages/pyanaconda/packaging/yumpayload.py", line 411, in updateBaseRepo
:    self._configureBaseRepo(self.storage, checkmount=checkmount)
:  File "/usr/lib64/python2.7/site-packages/pyanaconda/ui/gui/spokes/source.py", line 531, in getRepoMetadata
:    self.payload.updateBaseRepo(fallback=False, checkmount=False)
:  File "/usr/lib64/python2.7/threading.py", line 504, in run
:    self.__target(*self.__args, **self.__kwargs)
:  File "/usr/lib64/python2.7/site-packages/pyanaconda/threads.py", line 91, in run
:    threading.Thread.run(self, *args, **kwargs)
:OSError: [Errno 2] No such file or directory: '/run/install/repo/Fedora-18-Beta-x86_64-DVD.iso'

Comment 1 Adam Williamson 2012-11-22 10:05:22 UTC
Created attachment 649611 [details]
File: anaconda-tb

Comment 2 Adam Williamson 2012-11-22 10:05:24 UTC
Created attachment 649612 [details]
File: product

Comment 3 Adam Williamson 2012-11-22 10:05:26 UTC
Created attachment 649613 [details]
File: type

Comment 4 Adam Williamson 2012-11-22 10:05:28 UTC
Created attachment 649614 [details]
File: ifcfg.log

Comment 5 Adam Williamson 2012-11-22 10:05:29 UTC
Created attachment 649615 [details]
File: storage.log

Comment 6 Adam Williamson 2012-11-22 10:05:33 UTC
Created attachment 649616 [details]
File: version

Comment 7 Adam Williamson 2012-11-22 10:05:35 UTC
Created attachment 649617 [details]
File: environ

Comment 8 Adam Williamson 2012-11-22 10:05:36 UTC
Created attachment 649618 [details]
File: executable

Comment 9 Adam Williamson 2012-11-22 10:05:37 UTC
Created attachment 649619 [details]
File: anaconda.log

Comment 10 Adam Williamson 2012-11-22 10:05:39 UTC
Created attachment 649620 [details]
File: syslog

Comment 11 Adam Williamson 2012-11-22 10:05:41 UTC
Created attachment 649621 [details]
File: hashmarkername

Comment 12 Adam Williamson 2012-11-22 10:05:42 UTC
Created attachment 649622 [details]
File: packaging.log

Comment 13 Adam Williamson 2012-11-22 10:05:44 UTC
Created attachment 649623 [details]
File: cmdline_file

Comment 14 Adam Williamson 2012-11-22 10:05:45 UTC
Created attachment 649624 [details]
File: release

Comment 15 Adam Williamson 2012-11-22 10:05:47 UTC
Created attachment 649625 [details]
File: program.log

Comment 16 Adam Williamson 2012-11-22 10:10:02 UTC
I may be just dumb and/or drunk, but I've spent the last hour trying to get an NFS install of Beta RC1 to work and just cannot do it. It either ignores my NFS repo and goes to 'nearest mirror' or hits this TB.

I have an F17 system - 192.168.1.11 - with this /etc/exports :

/export 192.168.1.0/24(ro,sync,insecure,root_squash,no_subtree_check,fsid=0)
/export/fedora 192.168.1.0/24(rw,nohide,sync,insecure,root_squash,no_subtree_check)

Fedora-18-Beta-x86_64-DVD.iso is in the /export directory .

I hit this TB by booting netinst with default parameters, and at the installation source screen, selecting NFS from the drop-down and entering:

192.168.1.11:/Fedora-18-Beta-x86_64-DVD.iso

in the text box, then hitting Done. If I just enter:

192.168.1.11:/

then I get no tb, but it reports an error and the spoke shows as greyed out.

Various attempts at inst.repo=nfs:... incantations result in one of these three results - traceback, error/greyed out, or 'nearest mirror'.

If I run 'mount -t nfs 192.168.1.11:/ /mnt/temp' on my desktop, the mount works fine, and I see the .iso file in /mnt/temp . So the server appears to be correctly configured.

Proposing as a blocker, do please advise if this is, in some way, operator error.

Comment 17 Kamil Páral 2012-11-22 11:06:36 UTC
I just booted Beta RC1 netinst (anaconda 18.29.2) with inst.repo=nfs:192.168.1.1:/mnt/test , where /mnt/test contains loopback-mounted DVD. The Closest Mirror is selected by default.

Comment 18 Adam Williamson 2012-11-22 12:39:13 UTC
So...I managed to get a couple of successes by dropping the inst.stage2 parameter from the netinst parameters and passing inst.repo=nfsiso:blah. I cannot get it to work interactively, or while the inst.stage2 parameter is present.

Comment 19 Adam Williamson 2012-11-22 13:09:21 UTC
OK, NFS is definitely badly regressed in RC1 over TC9 :( This may be related to the fix for https://bugzilla.redhat.com/show_bug.cgi?id=876223 .

To summarize:

In TC9, I can append inst.repo=nfs or inst.repo=nfsiso on the command line just fine. No need to wipe the inst.stage2 parameter. They work interchangeably, btw, either will work for either type of NFS repo. I can pick an NFS repo from the Installation Source spoke (interactively) just fine (whether it's an exploded tree or just a directory with a .ISO file in it). Basically everything works fine *except* the single case mentioned in 876223 - where you explicitly specify a .iso filename.

In RC1, interactive selection of NFS repos of any kind - exploded tree or ISO, whether passing a directory or a precise .ISO path - appears broken. Just doesn't work. You wind up with a tb or Closest Mirror. Appending the inst.repo=nfs(iso) parameter also doesn't work (you get Closest Mirror). The only way to make it work is to delete the inst.stage2 parameter and replace it with inst.repo=nfs(iso).

I'll attach logs from a few failure cases.

Comment 20 Adam Williamson 2012-11-22 13:18:01 UTC
Created attachment 649795 [details]
packaging.log: appended repo=nfsiso:192.168.1.11:/export/fedora (result: Closest Mirror)

Comment 21 Adam Williamson 2012-11-22 13:19:03 UTC
Created attachment 649796 [details]
program.log: appended repo=nfsiso:192.168.1.11:/export/fedora (result: Closest Mirror)

Comment 22 Adam Williamson 2012-11-22 13:20:05 UTC
Created attachment 649798 [details]
syslog: appended repo=nfsiso:192.168.1.11:/export/fedora (result: Closest Mirror)

Comment 23 Adam Williamson 2012-11-22 13:21:27 UTC
Created attachment 649799 [details]
anaconda.log: appended repo=nfsiso:192.168.1.11:/export/fedora (result: Closest Mirror)

Comment 24 Kamil Páral 2012-11-22 13:27:41 UTC
I can confirm that the problem in comment 17 is solved if I remove inst.stage2= and provide the same inst.repo= as before.

Comment 25 Adam Williamson 2012-11-22 13:32:59 UTC
Created attachment 649802 [details]
packaging.log: booted netinst as usual, interactively specified NFS repo (result: Error setting up software source)

Comment 26 Adam Williamson 2012-11-22 13:34:01 UTC
Created attachment 649803 [details]
program.log: booted netinst as usual, interactively specified NFS repo (result: Error setting up software source)

Comment 27 Adam Williamson 2012-11-22 13:34:57 UTC
Created attachment 649804 [details]
anaconda.log: booted netinst as usual, interactively specified NFS repo (result: Error setting up software source)

Comment 28 Adam Williamson 2012-11-22 13:35:58 UTC
Created attachment 649805 [details]
syslog: booted netinst as usual, interactively specified NFS repo (result: Error setting up software source)

Comment 29 Kamil Páral 2012-11-22 17:45:16 UTC
I can also confirm (In reply to comment #24)
> I can confirm that the problem in comment 17 is solved if I remove
> inst.stage2= and provide the same inst.repo= as before.

The other workaround is to use direct kernel boot instead of netinst (that effectively doesn't set any inst.stage2= and therefore works OK). Applies for both NFS and NFSISO.

Comment 30 Adam Williamson 2012-11-22 18:43:33 UTC
For blocker reference, the relevant criterion here is "The installer must be able to use the HTTP, FTP and either NFS or NFSISO remote package source options". It does not explicitly state whether both cmdline and interactive, or just one of the two, should work.

Comment 31 Adam Williamson 2012-11-22 19:03:19 UTC
I believe http://git.fedorahosted.org/cgit/anaconda.git/commit/?h=f18-beta2-branch&id=1b7e6541b8ca5dbf7f76aa20a25837906ba2b2c5 is what broke this, BTW. Looking at the IRC logs, I remember why this was committed to Beta branch now. jlk was fixing 876897 , and his fix for that depended on the changes in this commit. dlehman actually ran it past me at the time:

20-11-2012 11:37:49 < dlehman!~dlehman.72.173: adamw: I know this probably
 seems like a mess, but when you have 8 patches in the works you can't always te
st them in isolation from each other and then again for every possible subset
20-11-2012 11:49:17 > adamw: dlehman: sure, i get that. go ahead and put that fix in.

I don't believe I actually eyeballed the patch, and honestly I'm not sure I'd have wound up the alarm if I had, I usually trust jlk's/dlehman's reads. So we weren't really going against policy here, the commit was a dep of an NTH fix, just unfortunate. I'll attach a full IRC log extract for forensic purposes.

Comment 32 Adam Williamson 2012-11-22 19:05:30 UTC
Created attachment 649998 [details]
IRC log extract from when we discussed applying this patch (for forensic purposes)

Comment 33 Kamil Páral 2012-11-22 19:08:04 UTC
*** Bug 879338 has been marked as a duplicate of this bug. ***

Comment 34 Adam Williamson 2012-11-22 20:49:21 UTC
Discussed at 2012-11-22 go/no-go meeting (http://meetbot.fedoraproject.org/fedora-meeting-1/2012-11-22/f18_beta_gono-go_meeting.2012-11-22-20.01.log.txt ) acting as a blocker review meeting. This is narrowly rejected as a blocker: it sucks, but there are two viable and documentable workarounds (replace inst.stage2= , and use direct kernel boot). Adding to CommonBugs, of course.

Comment 35 Kamil Páral 2012-11-22 21:02:24 UTC
Moving to final blocker instead.

Comment 36 Paul Franklin (RHlists) 2012-11-22 21:59:18 UTC
As I have said on other bugs, I am not a Fedora expert.

I am the author of 879338 which is referenced in Comment 33.

In that bug I used PXE to try to install RC1; it failed,
so if that is using "direct kernel boot" I respectfully
disagree that the workaround works.  Of course, if "direct
kernel boot" means something else then I apologize for
wasting everybody's time reading this comment.  (My PXE
command line is in that bug report; I had no stage2=...)
Thanks.

Comment 37 Adam Williamson 2012-11-22 23:02:25 UTC
It's possible your bug is something different, perhaps the use of UDP. Kamil just duped it on general principles. I'll check the logs and see.

Comment 38 Paul Franklin (RHlists) 2012-11-29 20:00:52 UTC
I'm confused why I don't see this bug listed on
http://qa.fedoraproject.org/blockerbugs/milestone/18/final/buglist
since it seems to say above "Blocks: F18Blocker" -- which I was
guessing meant it would be on that list.  Thanks.

Comment 39 Kamil Páral 2012-11-29 22:24:54 UTC
(In reply to comment #38)
Because I failed to remove RejectedBlocker from the Whiteboard. Thanks, fixed.

Comment 40 Tim Flink 2012-11-30 18:02:53 UTC
+1 final blocker due to violation of the following F18 beta release criterion [1]:

The installer must be able to use the HTTP, FTP and either NFS or NFSISO remote package source options.

[1] http://fedoraproject.org/wiki/Fedora_18_Beta_Release_Criteria

Comment 41 Adam Williamson 2012-12-01 08:14:31 UTC
+1 blocker for final, the workarounds are okay for Beta but not for Final.

Comment 42 Paul Franklin (RHlists) 2012-12-01 15:26:34 UTC
It never occurred to me it wouldn't be a final blocker,
but if we have to vote to make it so of course I vote
+1 to make it a final blocker, on the grounds of "The
installer must be able to use all supported local and
remote package source options."  (I thought it was, already.)

Comment 43 Adam Williamson 2012-12-01 19:02:07 UTC
all blockers are nominated, voted on and then accepted.

Comment 44 Jaroslav Reznik 2012-12-03 13:32:49 UTC
+1 final blocker, not a very nice workaround known

Comment 45 Tim Flink 2012-12-04 20:51:55 UTC
That makes for +4 blocker - moving to accepted

Comment 46 Chris Lumens 2012-12-04 22:02:56 UTC
I don't know what it means, but the initial traceback is suspiciously close to the one in bug 879612.

Comment 47 Brian Lane 2012-12-07 00:39:10 UTC
Please retest with smoke4, I've done about a dozen installs using inst.repo=nfs:IP:path and other than not rebooting at the end they installed just fine.

Comment 48 Kamil Páral 2012-12-07 10:34:20 UTC
I tried to verify bug 879187. I booted netinst and then selected installation source dialog and provided nfsiso url nfs:ip:/path/dvd.iso

Package: anaconda-18.36
OS Release: Fedora release 18

Comment 49 Kamil Páral 2012-12-07 10:35:26 UTC
Created attachment 659293 [details]
18.36 anaconda-tb for graphical nfsiso

Comment 50 Kamil Páral 2012-12-07 10:47:00 UTC
Created attachment 659296 [details]
18.36 anaconda-tb for cmdline nfsiso

Graphically I wasn't able to select nfs (not nfsiso) at all, it complained about invalid url all the time. There was no mention in packaging.log that it would try to mount anything. Manually I could mount it just fine.

The same applies for inst.repo=nfs:ip:path, closest mirror is selected after anaconda start and there is no mention in any logs of nfs mounting.

For inst.repo=nfs:ip:path/dvd.iso anaconda crashes immediately on start, traceback attached.

All tested with smoke4 netinst and smoke4 dvd as a nfs/nfsiso repo.

Comment 51 Samantha N. Bueno 2012-12-10 15:36:49 UTC
I still see the issue occurring in F18-TC1 (anaconda 18.36); hopefully I'll have a fix later today or tomorrow.

Comment 52 Robert Lightfoot 2012-12-11 13:29:46 UTC
QA Test Case NFSISO install F18-TC1

Booted from DVD and changed source to NFS iso file.


Package: anaconda-18.36
Architecture: i686
OS Release: Fedora release 18-TC1

Comment 53 Robert Lightfoot 2012-12-11 14:15:30 UTC
QA Test Case NFSISO Testing F18-TC1

Failed when trying read iso


Package: anaconda-18.36
OS Release: Fedora release 18-TC1

Comment 54 Jaroslav Reznik 2012-12-17 11:26:54 UTC
(In reply to comment #51)
> I still see the issue occurring in F18-TC1 (anaconda 18.36); hopefully I'll
> have a fix later today or tomorrow.

Samantha - any progress?

Comment 55 Samantha N. Bueno 2012-12-17 17:04:37 UTC
(In reply to comment #54)
> (In reply to comment #51)
> > I still see the issue occurring in F18-TC1 (anaconda 18.36); hopefully I'll
> > have a fix later today or tomorrow.
> 
> Samantha - any progress?

Sorry - this got preempted in favor of some RHEL issues. Hoping to take a look at it this week.

Comment 56 Samantha N. Bueno 2012-12-20 14:34:46 UTC
Still having to work on RHEL issues, so my apologies. I should mention that with F18-TC3 (anaconda 18.37.3), I'm not getting any tracebacks. Installation over interactive NFS fails, but the installer doesn't crash, and an error message is displayed.

As far as what exactly is causing the issue here though, the NFS repo (exploded tree or ISO) is never actually getting mounted. It always gets (incorrectly) detected as being mounted though....hence why anaconda continues to run, only failing when it finally hits a wall trying to access info that would be obtained from files in a mounted tree or ISO.

The issue does lie in the code from the patch linked in comment 31, but it's not possible to simply revert that at this point--too much has changed in the codebase since that was originally posted.

Comment 57 Fedora Update System 2012-12-21 01:28:57 UTC
anaconda-18.37.6-1.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/anaconda-18.37.6-1.fc18

Comment 58 Kamil Páral 2012-12-21 13:38:20 UTC
I tested anaconda-18.37.6-1.fc18 with:
* exploded tree in interactive mode
* dvd.iso in interactive mode
* exploded tree using inst.repo
* dvd.iso using inst.repo

Everything works. Except that nfsiso fails to reboot, which is bug 876218.

Comment 59 Fedora Update System 2013-01-02 21:48:24 UTC
dracut-024-17.git20121220.fc18, anaconda-18.37.8-1.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.