This killer-bug (It renders installing fedora impossible) is still present in Fedora-Xfce-Live-i386-26_Beta-1.4.iso.

<rant>
Whom do I need to bribe until somebody @RH finally reacts and fixes this bug?
</rant>

I would say this should be assigned to LiveCD and not to lvm...

(In reply to Milan Broz from comment #2)
> I would say this should be assigned to LiveCD and not to lvm...
I do not agree. This probably affects all Fedora i386 isos not only Xfce live.

This time I tried Fedora-Xfce-Live-i386-26_Beta-1.4.iso and Fedora-Workstation-Live-i386-26_Beta-1.4.iso both exhibit the same problem.

May-be I have been clear enough about the nastiness of this bug:
When booting from *.iso systemd hangs for 50 mins until it finally fails.

i686 live images are booting fine in openQA:

https://openqa.fedoraproject.org/tests/overview?distri=fedora&version=26&build=Fedora-26-20170531.0&groupid=1

(In reply to Adam Williamson from comment #4)
> i686 live images are booting fine in openQA:

Fedora-Workstation-Live-i386-26_Beta-1.4.iso does not boot for me.

As you don't seem to wanting to believe me, I will try to video tape it and to provide detailed logs.

It's not about believing you or not, it's comparing results: clearly there's some difference between the scenarios, which could help pinpoint the problem.

Here is the video, cut into 2 parts (please excuse the poor quality) and the corresponding rdsosreport.txt.

- https://corsepiu.fedorapeople.org/misc/RHBZ%231436096-20170602_1.mp4
is first part of a videotape showing booting Fedora-Workstation-Live-i386-26_Beta-1.4.iso on "gunvald1"[1]:

You can see the rpcbind warning mentioned about and how booting is hanging presumably due to some device-mapper related service misbehaving for 50mins.


- https://corsepiu.fedorapeople.org/misc/RHBZ%231436096-20170602_2.mp4
is the second part (starting at ca. 49mins of "hanging").

It shows how booting fails in the end.


- https://corsepiu.fedorapeople.org/misc/RHBZ%231436096-20170602-rdsosreport.txt
is the rdsosreport.txt from the bootup attempt depicted in the videos.

It contains lines, I interpret as dracut-segfaults:
$ grep -i segfault rdsosreport.txt
[   12.708980] localhost kernel: dracut-cmdline[177]: segfault at 82177000 ip b76e5239 sp bfa7f2e8 error 4 in libc-2.25.so[b7578000+1dd000]
[   13.296646] localhost kernel: dracut-pre-trig[269]: segfault at 815dc000 ip b765c239 sp bf902198 error 4 in libc-2.25.so[b74ef000+1dd000]

It contains lines, I interpret as rpcbind (and may-be systemd) failures + followups of these failures:
$ grep -i failed rdsosreport.txt
[   12.709309] localhost kernel: Core dump to |/usr/lib/systemd/systemd-coredump 177 0 0 11 1496380640 4294967295 dracut-cmdline pipe failed
[   13.023968] localhost systemd-tmpfiles[240]: Failed to open 'rpcbind.conf': No such file or directory
[   13.089651] localhost rpc.statd[244]: Failed to register (statd, 1, udp): svc_reg() err: RPC: Remote system error - Connection refused
[   13.105825] localhost rpc.statd[244]: Failed to register (statd, 1, tcp): svc_reg() err: RPC: Remote system error - Connection refused
[   13.122251] localhost rpc.statd[244]: Failed to register (statd, 1, udp6): svc_reg() err: RPC: Remote system error - Connection refused
[   13.138515] localhost rpc.statd[244]: Failed to register (statd, 1, tcp6): svc_reg() err: RPC: Remote system error - Connection refused
[   13.139720] localhost rpc.statd[244]: failed to create RPC listeners, exiting
[   13.297458] localhost kernel: Core dump to |/usr/lib/systemd/systemd-coredump 269 0 0 11 1496380641 4294967295 dracut-pre-trig pipe failed
[ 3012.627830] localhost systemd[1]: Dependency failed for /sysroot.
[ 3012.628931] localhost systemd[1]: Dependency failed for Initrd Root File System.
[ 3012.630177] localhost systemd[1]: Dependency failed for Reload Configuration from the Real Root.
[ 3012.631276] localhost systemd[1]: initrd-parse-etc.service: Job initrd-parse-etc.service/start failed with result 'dependency'.
[ 3012.645101] localhost systemd[1]: initrd-root-fs.target: Job initrd-root-fs.target/start failed with result 'dependency'.
[ 3012.659237] localhost systemd[1]: sysroot.mount: Job sysroot.mount/start failed with result 'dependency'.
[ 3012.661141] localhost systemd[1]: dev-mapper-live\x2drw.device: Job dev-mapper-live\x2drw.device/start failed with result 'timeout'.



[1] gunvald1: A Medion Akoya E1210 netbook from 2008, with 2GB RAM, Intel Atom N270 CPU. It currently is configured in a fairly complex grub/BIOS-multiboot configuration[2] hosting Win10, Debian 8, several Fedoras and several "data" partitions. In its history, it had hosted various Fedora, openSUSE and Ubuntu releases, as well as WinXP, Win8, Win8.1 and Win10.

From /proc/cpuinfo:
processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 28
model name	: Intel(R) Atom(TM) CPU N270   @ 1.60GHz
stepping	: 2
...
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx constant_tsc arch_perfmon pebs bts aperfmperf pni dtes64 monitor ds_cpl est tm2 ssse3 xtpr pdcm movbe lahf_lm dtherm

[2] I don't think this is of any importance here. I can reproduce this BZ on another, Win10-only netbook (no grub nor any other piece of linux on at all).

(In reply to Ralf Corsepius from comment #7)
> Here is the video, cut into 2 parts (please excuse the poor quality) and the
> corresponding rdsosreport.txt.

Hooray! Now it looks like data. No idea how you come with video idea, screenshots are often usually worthless, video same but taking much more space... rdsosreport is the valuable thing.

> https://corsepiu.fedorapeople.org/misc/RHBZ%231436096-20170602-rdsosreport.
> txt
> is the rdsosreport.txt from the bootup attempt depicted in the videos.
> 
> It contains lines, I interpret as dracut-segfaults:
> $ grep -i segfault rdsosreport.txt
> [   12.708980] localhost kernel: dracut-cmdline[177]: segfault at 82177000
> ip b76e5239 sp bfa7f2e8 error 4 in libc-2.25.so[b7578000+1dd000]
> [   13.296646] localhost kernel: dracut-pre-trig[269]: segfault at 815dc000
> ip b765c239 sp bf902198 error 4 in libc-2.25.so[b74ef000+1dd000]
> 
> It contains lines, I interpret as rpcbind (and may-be systemd) failures +
> followups of these failures:
...

If there are serious problems with dracut no one can reproduce and you want them fixed, you will have to do the work, no need to bribe anyone.

If you could gdb the coredumps or post them instead of the vidoes that would be great.

Remove the `quiet` from kernel options and add `rd.debug systemd.log_level=debug systemd.debug_shell=1`

While it is waiting for something, try Alt+SysRq+w (and with m, l, d, q) to get list of blocking processes, memory alloc,...

Also you should be able to get to the debug shell using Ctrl+Alt+F9 and investigate...

Once in emergency shell, what does systemctl say? Any failed services?

Check systemctl status $SERVICE of all failed units.

...

More:

https://en.wikipedia.org/wiki/Magic_SysRq_key
https://fedoraproject.org/wiki/How_to_debug_Dracut_problems#Debugging
https://www.freedesktop.org/wiki/Software/systemd/Debugging/

(In reply to Marian Csontos from comment #8)
> (In reply to Ralf Corsepius from comment #7)
> > Here is the video, cut into 2 parts (please excuse the poor quality) and the
> > corresponding rdsosreport.txt.
> 
> Hooray! Now it looks like data. No idea how you come with video idea,
I learned such stuff occasionally are inevitable, because there are people who prefer to ignore bugs, even though they have been reported months (the rpcbind bug is knows for many months) until they see them personally 
or don't grasp the impact the impact their bugs have (fedora-26-i686 is unusable).

(In reply to Marian Csontos from comment #8)
> Once in emergency shell, what does systemctl say? Any failed services?
> 
> Check systemctl status $SERVICE of all failed units.
Is this supposed to work in emergency mode?

Whatever I try, systemctl returns "Bad message"

I don't see this when I attempt to install from Fedora-Workstation-Live-i386-26_Beta-1.4.iso

Marian: no-one has tried super hard to reproduce this yet, in all fairness. I think one significant difference is that no-one else is trying on an actual i686 CPU; openQA tests the i686 images on a 64-bit virtual CPU, and I think Mike was also testing with a 64-bit CPU. One thing we could do is send out a mail to test@ and devel@ asking if folks with actual 32-bit CPUs see the same thing Ralf does.

Unfortunately I no longer own any true 32-bit CPUs, so I can't check that :/

Do note that i686 was removed from the list of release-blocking arches for Fedora a few cycles back, which was an intentional downgrade on the basis that it's known to be not terribly heavily supported in some upstream projects - e.g. the kernel - any more. So stuff like this *is* going to happen...the other non-blocking arches (ppc64, aarch64 for e.g.) have quite active specialist communities who look into arch-specific issues like this, but AFAIK there really isn't an 'i686 group' yet. It may be time for folks who still use i686 to form one...

(In reply to Adam Williamson from comment #12)
> Marian: no-one has tried super hard to reproduce this yet, in all fairness.
> I think one significant difference is that no-one else is trying on an
> actual i686 CPU; openQA tests the i686 images on a 64-bit virtual CPU, and I
> think Mike was also testing with a 64-bit CPU. One thing we could do is send
> out a mail to test@ and devel@ asking if folks with actual 32-bit CPUs see
> the same thing Ralf does.
FWIW: I meanwhile tried Fedora-Workstation-Live-i386-26_Beta-1.4.iso on x86_64/BIOS system. It did not expose these problems.

> Unfortunately I no longer own any true 32-bit CPUs, so I can't check that :/
I still have 3. 2 expose these problems. 

The 3rd is a original Pentium III (from ca. 2001), which currently has fc25 installed, but which IIRC, lacks some of the cpu-features required for fc26 and is very hard to be used for testing (No USB, No DVD). It very likely will be "dumped" when Fc25 goes EOL because it isn't in actual use anymore.

> Do note that i686 was removed from the list of release-blocking arches for
> Fedora a few cycles back, which was an intentional downgrade on the basis
> that it's known to be not terribly heavily supported in some upstream
> projects - e.g. the kernel - any more.
Well, ... over the years, we all have learned Fedora/RH business goals/management objectives don't necessarily match "the community's"

> So stuff like this *is* going to
> happen...the other non-blocking arches (ppc64, aarch64 for e.g.) have quite
> active specialist communities who look into arch-specific issues like this,
> but AFAIK there really isn't an 'i686 group' yet. It may be time for folks
> who still use i686 to form one...
Likely the group of people using i686ers hasn't noticed the harm which has been going on in Fedora, because fc25 so far has worked mostly flawless on i686ers, while the other archs basically are toys with a hardly measurable userbase, IMHO. 

That being said, my way of fixing bugs for "secondary archs" but the i686, will be to excluded this package on affected archs.

Created attachment 1288531 [details]
systemd-udevd i686 crash

I also see this happening with the F26 XFCE Live Beta (i386) on two netbooks: one with an Atom N270 CPU, the other one with an Atom N450 CPU.  N270 is 32-bit only, but N450 supports 64-bit extensions and it's normally running F25 XFCE x86_64.  F26 XFCE Live Beta x86_64 works fine on the N450 netbook.

I booted the i386 image with "rd.debug systemd.log_level=debug systemd.debug_shell=1", as suggested by comment 8.  SysRq-w only produces the message "This SysRq function in not supported".  During the 50 minutes wait time I saw systemd-udevd repeatedly crashing (see the attached screenshot).  After 50 minutes, I saw the debug shell starting and informing me where I can find rdsosreport.txt.  Unfortunately, I couldn't do anything there: after just a few seconds, another message appeared about restarting systemd-udevd and the netbook stopped responding to the keyboard.  Not even Ctrl-Alt-Del worked any more (this was still functional during the 50 mins wait time, during previous boot attempts), and short presses of the power button also had no effect.  I had to press the power button for 4 seconds to cut power.

Neither netbook has a serial port - if there's anything I can do to help with debugging, please let me know.

The same F26 XFCE Live Beta i386 works on a Dell Vostro 260 desktop (with an Intel Core i3 CPU).

Perhaps this bug only happens on Atom CPUs? I remember an announcement about changing the target CPU for gcc, but I would expect systemd-udevd to die with SIGILL(4), not SIGABRT(6) as in the screenshot.

(In reply to Laurentiu Pancescu from comment #15)
> The same F26 XFCE Live Beta i386 works on a Dell Vostro 260 desktop (with an
> Intel Core i3 CPU).
Is this an ix86 CPU? AFAICT, all Intel i3's are 64bit CPUs.

> Perhaps this bug only happens on Atom CPUs?
I don't know, but I doubt it, because an fc25 installation with most packages upgraded to fc26 works.

(In reply to Ralf Corsepius from comment #16)
> Is this an ix86 CPU? AFAICT, all Intel i3's are 64bit CPUs.

That's right, i3 supports 64-bit.  But so is Atom N450, which is also 64-bit, and it produces the hang you reported.  Comment 12 formulated the hypothesis that only 32-bit CPUs have problems with F26, and I think Atom N450 proves this wrong.  I was curious if I can reproduce it on another 64-bit CPU, or it's perhaps specific to Atoms.  I was suspecting that the gcc generated code might break on Atom CPUs, if they target other a higher x86 revision than before.
 
> > Perhaps this bug only happens on Atom CPUs?
> I don't know, but I doubt it, because an fc25 installation with most
> packages upgraded to fc26 works.

Was it also systemd-udevd on your hardware?  Did you also upgrade systemd?  I didn't try to upgrade from F25 to F26 Beta with dnf system-upgrade and see if it hangs on regular boots.

'Atom' is really just a branding name they've applied to various quite different CPU cores over the years, so I'd be surprised if it's as simple as "it affects anything called Atom". But it seems pretty clear there's some relationship to CPU capabilities, here. It might be useful to have the /proc/cpuinfo contents for all known affected CPUs, and 'interesting' non-affected CPUs (ones we thought might be affected, but which actually aren't...)

(In reply to Laurentiu Pancescu from comment #17)
> (In reply to Ralf Corsepius from comment #16)
> > Is this an ix86 CPU? AFAICT, all Intel i3's are 64bit CPUs.
> 
> That's right, i3 supports 64-bit.  But so is Atom N450, which is also
> 64-bit, and it produces the hang you reported.
This is an N270, a 32 bit Atom-CPU (see Comment 7)

> Comment 12 formulated the
> hypothesis that only 32-bit CPUs have problems with F26, and I think Atom
> N450 proves this wrong.
Like I said (see Comment 17), I did not have any probs with running 32-bit Fedora 26 on 64-bit CPUs. My problems were with running/installing 32-bit Fedora-26 on an 32-bit Atom.

> > > Perhaps this bug only happens on Atom CPUs?
> > I don't know, but I doubt it, because an fc25 installation with most
> > packages upgraded to fc26 works.
> 
> Was it also systemd-udevd on your hardware?  Did you also upgrade systemd?
Until yesterday, I had not been able to install glibc-2.25 (and thus f26's systemd and other packages which require glibc-2.25) on this system, ever since Fedora 26 development was launched.

However, yesterday, to my surprise, it finally succeeded! I am still observing a segfault (from lvmetad) during very early stages of booting, nevertheless the system comes up in the end and appears to "work".

Also, Fedora-Workstation-netinst-i386-26-20170619.n.0.iso is not immediately crashing anymore (the rpcbind warnings are still there) and the installer is coming up!

No idea about the cause. To me, it appears as if some changes between beta 1.4 and 20170619 seem to have remedied the worst issues, I had.

> I didn't try to upgrade from F25 to F26 Beta with dnf system-upgrade and see
> if it hangs on regular boots.
I haven't tried this in recent weeks. Attempts in Apr/May and before were not successful.

Created attachment 1289815 [details]
/proc/cpuinfo from the Atom N270 netbook

I attached the cpuinfo from the Atom N270 netbook, as suggested by Adam Williamson in comment 18 (I'll add the N450 cpuinfo after I get home).

Created attachment 1289849 [details]
/proc/cpuinfo from the Atom N450 netbook

(In reply to Ralf Corsepius from comment #19)
> This is an N270, a 32 bit Atom-CPU (see Comment 7)

I know, this is what made me curious. I saw i386 F26 hang with both Atom N270 (32-bit) and N450 (64-bit), but not the Core i3 (also 64-bit).

Created attachment 1290025 [details]
/proc/cpuinfo for Intel Core i3 (the Dell Vostro 260 desktop)

(In reply to Ralf Corsepius from comment #19)
> However, yesterday, to my surprise, it finally succeeded! I am still
> observing a segfault (from lvmetad) during very early stages of booting,
> nevertheless the system comes up in the end and appears to "work".

This update
https://bodhi.fedoraproject.org/updates/FEDORA-2017-12c8e8e75c
to udisks2 seems to fix this segfault for me.

Time to close this BZ, I think.

The latest nightly from 2017-06-22 boots successfully (I see a new error message two times, "dracut-pre-udev[329]: rpc.idmapd: conf_reinit: open("(null)", O_RDONLY) failed", but it's before looking for the boot media).  So it does seem to be fixed.

https://kojipkgs.fedoraproject.org/compose/branched/Fedora-26-20170622.n.0/compose/Spins/i386/iso/Fedora-Xfce-Live-i386-26-20170622.n.0.iso

Sounds like we can close this. Thanks for the updates, guys.

rpc.idmapd: conf_reinit: open
("(null)", O_RDONLY) failed
(In reply to Laurentiu Pancescu from comment #25)
> The latest nightly from 2017-06-22 boots successfully (I see a new error
> message two times, "dracut-pre-udev[329]: rpc.idmapd: conf_reinit:
> open("(null)", O_RDONLY) failed", but it's before looking for the boot
> media).  So it does seem to be fixed.
> 
> https://kojipkgs.fedoraproject.org/compose/branched/Fedora-26-20170622.n.0/
> compose/Spins/i386/iso/Fedora-Xfce-Live-i386-26-20170622.n.0.iso

I'm getting this now on Fedora 25 I posted a Bugzilla https://bugzilla.redhat.com/show_bug.cgi?id=1466850

nfs-utils x86_64 1:2.1.1-5.rc4