Bug 853508

Created attachment 608611 [details]
copy of dracut-state.sh

Created attachment 608612 [details]
output of journalctl -a

Created attachment 608615 [details]
output of journalctl --output=verbose

Created attachment 608616 [details]
copy of system.journal

Assuming your images aren't the very latest, this is very likely the same problem as bug 849672.

Could you fetch a copy of TC5 (or later) test images and see if that works? You should be able to get the images from a link off this page:
https://fedoraproject.org/wiki/Test_Results:Current_Installation_Test



By the way: if you're using a DVD image on NFS you shouldn't need to extract the squashfs.img. The anaconda code in initrd.img should be smart enough to mount the .iso and use the squashfs image out of there. It should work if you pass the directory name or the full path to the iso itself, e.g.:

  inst.{repo,stage2}=nfs:...:/path/containing/iso/
  inst.{repo,stage2}=nfs:...:/path/containing/iso/fedora-XXX.iso

That should save you a little time and trouble in the future - although I'd suggest you wait until you've confirmed that the TC5 image works as expected with the setup you're already using. Changing multiple variables at once isn't great science, after all..

Thank you for your response.

I don't really have a lot of time right now, since I am
leaving town tomorrow for about a week.  (While I hope to be
able to read my email, I'll only be taking a laptop along.)

Nonetheless, since your request (that I try TC5) was so
specific, I went to the library today and tried to download
it.  It started all right, with a report (from the stupid
WXP browser) that it should only take an hour and forty
minutes.  But as it went along it got slower and slower and
when I finally aborted it I was more than halfway through my
allotted two hours and only forty percent of it was done.

So if it is somehow absolutely positively necessary to you
that I download that DVD and then try to install it, when I
get back I will set up an rsync download (here, not at the
library, I can't run a shell there,) and wait a few days.

In case the test I performed before is good enough this time
too, I downloaded today's version of the boot.iso file from
http://mirrors.kernel.org/fedora/development/18/i386/os/images
(and then renamed it to 201209040700_boot.iso).  I then
looped it and extracted vmlinuz and initrd.img as usual.
Then put copies of them in my /tftpboot/pxe/F18/20120904.

Here is what "ls -lFR /home/me/F/F18/20120904" says:

/home/me/F/F18/20120904:
total 303256
-r--r--r--    1 me me  278921216 Sep  4 07:00 201209040700_boot.iso
-r--r--r--    1 me me  26062060 Sep  4 06:59 initrd.img
-r--r--r--    1 me me   5223184 Aug 22 07:43 vmlinuz

(I have no way to know whether those two are identical to
the ones in the full DVD image you wanted me to download.  I
assume so but of course you will know for sure.  I doubt
that your suggestion of outdated files is the problem, since
the 8/28 attempt I documented in this bug was merely one of
my attempts.  I tried the 1 September one too, say.  But I
tried today's anyway, just in case it matters.  It didn't.)

I have tried lots of things, including what I reported in
this report earlier.  Nothing has worked.  Whether or not I
unpacked the squashfs.img has never mattered.  I just don't
feel like documenting every failed attempt.

Here is part of my /tftpboot/pxe/pxelinux.cfg/default file:

label 170
  kernel F18/20120904/vmlinuz
  append initrd=F18/20120904/initrd.img ramdisk_size=10000 repo=nfs:proto=udp:172.23.103.50:/home/me/F/F18/20120904/ ip=dhcp noipv6 loglevel=debug keymap=us lang=en_US.UTF-8 selinux=0 nodmraid

label 171
  kernel F18/20120904/vmlinuz
  append initrd=F18/20120904/initrd.img ramdisk_size=10000 repo=http://mirrors.kernel.org/fedora/development/18/i386/os ip=dhcp noipv6 stage2=nfs:proto=udp:172.23.103.50:/home/me/F/F18/20120904/ loglevel=debug keymap=us lang=en_US.UTF-8 selinux=0 nodmraid

But as I mentioned, neither attempt worked.

The error typeout I saw did not seem to be different from
the ones I attached earlier so I am not going to attach any
today.  (As I mentioned I'm short on time.)  My guess is
that the necessary clues (whatever they are) are in the
files I attached earlier.  Somewhere.

If you still have no luck in discovering the problem, I will
be happy to assist you in finding it, when I return from my
trip.  (Even downloading whatever full DVD image you tell
me to, if you do insist upon it.)

I would request that in such a case you tell me the specific
PXE kernel arguments you want me to use (and I will modify
them for my environment) -- remembering that I do not want
to do a full net-install, I want a local squashfs.img or
boot.iso, etc.  (Eventually, of course I'll download the
full F18 DVD, the final one.)  Please tell me whatever files
(or command output) you then want me to attach, next time.

Thanks.

It will be interesting to see if my bug is fixed when this one is:
https://bugzilla.redhat.com/show_bug.cgi?id=854180

This bug may be relevant to 855170

Can anybody tell me when the boot.iso file (which I mentioned
in comment #6 above) will have an initrd.img with the updated
dracut-023-39.git20120910.fc18 in it (as mentioned in 855170)?

Today's (12 Sept) seems to still have the old dracut-023-21.

Thanks.

Probably after the Alpha is released.  This is not a alpha blocker issue.  When https://admin.fedoraproject.org/updates/FEDORA-2012-13731/dracut-023-39.git20120910.fc18  gets pushed to stable, the next day's compose should make use of it.

Created attachment 615474 [details]
dmesg output from try 178

Created attachment 615475 [details]
journalctl -a output from try 178

Created attachment 615476 [details]
copy of system.journal from try 180

Thank you for your response.  I especially appreciated the
pointer, which enabled me to follow that dracut's progress.

Unfortunately my tentative assumption is that is doesn't fix
my problem (whatever it may be).

To test it I downloaded 9/19's version of boot.iso from
http://mirrors.kernel.org/fedora/development/18/i386/os/images
(and then renamed it to 20120919_boot.iso).  I then looped
it and extracted vmlinuz and initrd.img as usual.  Then put
copies of them in my /tftpboot/pxe/F18/20120919.

Here is what "ls -lFR /home/me/F/F18/20120919" says:

/home/me/F/F18/20120904:
total 305780
-r--r--r--    1 me me  281018368 Sep 19 07:21 20120919_boot.iso
-r--r--r--    1 me me  26547024 Sep 19 07:19 initrd.img
-r--r--r--    1 me me   5223184 Aug 22 07:43 vmlinuz

Here is part of my /tftpboot/pxe/pxelinux.cfg/default file:

label 178
  kernel F18/20120919/vmlinuz
  append initrd=F18/20120919/initrd.img ramdisk_size=10000 repo=nfs:proto=udp:172.23.103.50:/home/me/F/F18/20120919/ ip=dhcp noipv6 loglevel=debug keymap=us lang=en_US.UTF-8 selinux=0 nodmraid

label 180
  kernel F18/20120919/vmlinuz
  append initrd=F18/20120919/initrd.img ramdisk_size=10000 repo=nfs:proto=udp:172.23.103.50:/home/me/F/F18/20120919/ ip=dhcp noipv6 loglevel=debug keymap=us lang=en_US.UTF-8 selinux=0 nodmraid rd.debug

As you can see, they are the same format as things I have
previously tried, only accessing the 9/19 initrd.img this
time.  (The second had the rd.debug flag turned on.)

Neither attempt worked.

It takes me a while to save things from that dracut shell
and then get them to where I can upload them, so I didn't do
it on 9/19, when I first tried it.  But I'm guessing it
won't matter to you.  If it does (and you want me to try the
9/21 initrd.img or wait for something to happen), say so.

As before, I don't really know what you may need in order to
debug this so I am attaching a few things.

If you have no luck in discovering the problem, I will be
happy to assist you in finding it.  Just let me know.

Thanks.

Proposing as Beta blocker:
"It must be possible to install by booting the installation
kernel directly, including via PXE, and correctly specifying
a remote source for the installer itself, using whichever
protocols are required to work for package retrieval at the
current phase (Alpha, Beta, Final). This must work if the
remote source is not a complete repository but contains only
the files necessary for the installer itself to run."

Discussed at 2012-10-04 blocker review meeting: http://meetbot.fedoraproject.org/fedora-qa/2012-10-04/f18-beta-blocker-review-2.1.2012-10-04-16.00.log.txt . We noted that Kamil Paral filed a pass for the PXE validation test for Beta TC1: https://fedoraproject.org/wiki/Test_Results:Fedora_18_Beta_TC1_Install#PXE_Images . So we need more data to know what's going on here exactly and if this should be a blocker.

Paul, can you compare your configuration to that in the test case - https://fedoraproject.org/wiki/QA:Testcase_Boot_Methods_Pxeboot - and see if there are significant differences? kparal, can you compare your test configuration to Paul's? Paul, can you re-test with Beta TC1 or TC2 - http://dl.fedoraproject.org/pub/alt/stage/ - and see if it is still broken for you? Thanks!

Just a quick note here - all my testing was done with inst.repo=http:// protocol. The bug here seems to be related to nfs protocol. That might be a game-changer here. I'll retest as soon as I can with nfs as well.

(In reply to comment #17)
> Just a quick note here - all my testing was done with inst.repo=http://
> protocol. The bug here seems to be related to nfs protocol. That might be a
> game-changer here. I'll retest as soon as I can with nfs as well.

inst.repo=nfs is broken until anaconda-18.13

I've tested pxe with:

LABEL F18
        kernel /images/f18/vmlinuz
        MENU LABEL f18-nfs-serial
        append initrd=/images/f18/initrd.img ksdevice=bootif kssendmac inst.repo=http://reducio/mirror/development/18/x86_64/os console=ttyS0
        ipappend 2

and that works.

append initrd=/images/f18/initrd.img ksdevice=bootif kssendmac inst.repo=nfs:reducio:/srv/mirror/development/18/x86_64/os console=ttyS0

works as well (provided I'm using the Beta TC2 image set (anaconda 18.12)).

These are the common methods of doing pxe installs, what more is not working?

Ok, looks like nfsiso support isn't working quite right, so that's something to look into.  Not sure if that's beta release requirement or not.

Ok, the anaconda dracut module that tries to mount the iso fails.  We've got the iso dir mounted at /run/install/repo, and when we detect that there is an iso within, we try to move that mount point:

mount --move /run/install/repo /run/install/isodir

This fails with:

mount: wrong fs type, bad option, bad superblock on /run/install/repo,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail or so

Chasing up with the kernel team why mount --move isn't working for us.

*** Bug 862996 has been marked as a duplicate of this bug. ***

> Paul, can you compare your configuration to that in the test case - https://fedoraproject.org/wiki/QA:Testcase_Boot_Methods_Pxeboot - and see if there are significant differences? 

Well, as I said in my first post I don't claim to be a Fedora expert
but when I looked at QA:Testcase_Boot_Methods_Pxeboot it seemed to be
aimed at just booting the kernel and initrd image, then having the
user interactively enter in their desired options.  As you can see, I
placed many of my options in the PXE's append line.  I assume there
is a different path through your software (but of course I am just
guessing).  Perhaps therein lies the difference?

Note also I am giving an option to nfs, so perhaps that is it?

Having read Comment #21 I am aborting the download I had
started of 18-Beta-TC2.  I will wait for further instructions
here before I try downloading anything.

However, as I said earlier I am willing to help, try things,
give you whatever info you ask for.  Etc.

Thank you all.  I appreciate it.

That test case needs to be updated, we don't have a facility to handle the "askmethod" type questions any more.  The path to stage2 has to be provided at boot time, or autodiscovered from the boot media (boot.iso/netinst.iso or the full DVD).

In the particular case you're testing, nfs, it will work provided you are pointing to a nfs tree that does have a LiveOS/squashfs.img path.  If you're pointing to a nfs tree that just has an iso it won't work due to the bug I just discovered.

jesse: PXE alone is a Beta requirement, and NFS ISO alone is also a Beta requirement, the combination of the two is uncharted territory =)

Ok, the base problem appears to be that /run/ is a "shared" mount, and we cannot do moves on shared mounts (see bug 847418).  A bodge would be to mount --make-private /run  before doing the mount --move call, then mount --shared /run again, but that's pretty ugly.

I'm adding Harald and Lennart to the CC list to get info from them about why /run/ is shared in the dracut environment to begin with, and see if there is any better way to accomplish what we're trying to do here, before I put in a bodge.

Discussed at 2012-10-10 blocker review meeting: http://meetbot.fedoraproject.org/fedora-qa/2012-10-10/f18beta-blocker-review-3.2012-10-10-16.05.log.txt . We agreed in principle that, at Beta, either NFS non-ISO or NFS ISO remote sources need to work, but not both. The current criteria just say 'NFS' and don't really specify what this means, but in the past we've interpreted it to require NFS ISO, we don't think this is really right.

Kamil will propose changes to the criteria to cover this. On the basis of the agreement, this bug is rejected as a blocker, as plain NFS does work (and NFS ISO can be made to work with a workaround - provide LiveOS/squashfs.img). Re-proposing for Final, and adding CommonBugs so we can document the workarounds if this isn't fixed by Beta time.

Ping Harald, Lennart, can you shed some light on what should be done here?

Humm, so we were asked to make everything shared, in order to play nice with containers.

Is there a reason why --move doesn't work, and can't be made work in the kernel? Any chance to replace this with --bind plus a umount?

From what I heard from the kernel folks, it was a simple "no, we're not going to make --move work on shared filesystems"

Anyway I've tried your suggestion of --bind and then umount, and that seems to work in testing so I've submitted a patch.  It's a little bit nicer than the other suggestion which was to mark the filesystem as non-shared, do the move, then re-mark it as shared.

This issue affects installation from harddrive(repo=hd:<device>:/path) too.

anaconda-18.17-1.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/anaconda-18.17-1.fc18

Package anaconda-18.17-1.fc18:
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing anaconda-18.17-1.fc18'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-16295/anaconda-18.17-1.fc18
then log in and leave karma (feedback).

anaconda-18.18-1.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/anaconda-18.18-1.fc18

Package anaconda-18.18-1.fc18:
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing anaconda-18.18-1.fc18'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-16402/anaconda-18.18-1.fc18
then log in and leave karma (feedback).

anaconda-18.19-1.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/anaconda-18.19-1.fc18

I have downloaded the 18-Beta-TC6 i386 DVD image,
in the hope it might have some updated anaconda.

It failed to boot, leaving me in the dracut shell.

If necessary I will generate a "journalctl -a" output
file, and upload it here, but in the hope you can see
the problem based on my reading it (and guessing), it
seems the problem is a wrong argument to some "mount"
command, since it says:

... dracut-initqueue[268]: mount: unrecognized option '--rprivate'

after which it gives a list of possible mount options.

So maybe '--make-rprivate' is what is needed, instead?

But as I've said before, I am no expert.  I'm guessing.

Let me know what you want me to do, upload some file,
run some command in the dracit shell, etc.  Thanks.

If I had to make a further guess, I'd guess:

.../pyanaconda/packaging/yumpayload.py

is where the offending code is:

iutil.execWithRedirect("mount",
                       ["--rprivate", "/"],
                       stderr="/dev/tty5", stdout="/dev/tty5")

but again I am just guessing.

I don't know how to patch the initramfs.img so I
will just wait for an updated anaconda, as usual.

Thanks.

Paul, how do you boot? Bare-metal or virtual? Optical disk, converted to USB?

TC6 DVD in KVM boots just fine for me.

Please ignore my last post, I just got confused here (thinking about a different bug).

I boot via PXE, on "bare metal" (real machines, one
a PXE server, one a PXE client, no USB, no optical drive),
with a downloaded ISO image (of some sort, it varies) on
the server machine, NFS-mounted on the client.

Yes, I've wondered how you guys keep all the bugzilla
messages separate, in your minds.  You must get dozens
every day.  I don't envy you.

I will say I was surprised when I saw the "rprivate"
error message, since I had guessed a bind/umount pair
would be used instead.  Oh well.  I'm not trying to
tell you guys what to do; I'm just trying to install
Fedora 18, in the same way I've always installed them.

Thanks.

There's a couple of invocations of 'mount --rprivate' in dracut/anaconda-lib.sh as well, they may be better candidates as they're in the dracut stuff.

Since this bug was originally filed because of different issue, which should be resolved, I created a new one for the -rprivate error: Bug 869246 .

It seems to me that "nfsiso source for stage2 does not work"
still summarizes the problem fairly accurately, even if it
originally did not work for another reason.

Clearly I do not agree that it has been "fixed in 18.17"
(see above), or that it has been "resolved" as you put it.

But if you people think there should be a new bug at least
I would hope that you would make it a blocker of some sort.

Thanks.

It's hard for me to follow things, as I am not used to
tracking changes in git, never mind in anaconda, but I
would guess that another argument for not having another
bug is that the yumpayload.py lines I reported above, in
Comment #38, were added in response to this bug report.

Thus any change to them can be thought of as a refinement
to the response to this particular bug.

I would have guessed that the dracut/anaconda-lib.sh lines
would have been the ones which required a different bug
to be created, not this bug, if anything.  Oh well.

The make-rprivate fixes will be in anaconda-19.20

http://git.fedorahosted.org/cgit/anaconda.git/commit/?id=af5c706832622c9eb158e6c994ce240776a0f817

Sorry.  Make that "18.20" instead.

This bugreport became quite unreadable. But I have tested booting in VM using vmlinuz+initrd+inst.repo=nfs:server:/path/dvd.iso and it booted and installed OK - F18 Beta TC8, anaconda 18.28. This was supposed to be fixed in 18.19, which is stable. So I'm closing this bug, and please reopen if nfsiso doesn't boot and install. For other issues please report separate bugs. Thanks.