Bug 987657

Summary: Beaker does not provision rhel7 due to dracut problem
Product: Red Hat Enterprise Linux 7 Reporter: Russ Anderson <rja>
Component: anacondaAssignee: Anaconda Maintenance Team <anaconda-maint-list>
Status: CLOSED NOTABUG QA Contact: Release Test Team <release-test-team-automation>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 7.0CC: ctatman, dracut-maint-list, gbeshers, gbeshers, harald, jdonohue, loriann, nhorman, peterm, randerso, rja, spoyarek, tee, wwoods
Target Milestone: rc   
Target Release: 7.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-09-18 12:29:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 920743    
Attachments:
Description Flags
Additional failure details.
none
kickstart contents
none
rdsosreport.txt with rd.debug none

Description Russ Anderson 2013-07-23 19:56:03 UTC
Description of problem:  When using Beaker to provision rhel7 on UV2, dracut complains "dracut Warning: /dev/root does not exist" and "Could not boot." It then stops at the dracut prompt.

The last lines were:
-----------------------------------------------
[   46.921628] dracut: Scanning for all btrfs devices
[   47.443514] dracut: Scanning for all btrfs devices
[   47.456978] dracut Warning: Could not boot.

           
dracut Warning: [   47.463714] dracut Warning: /dev/root does not exist
Could not boot.

Couldn't open /dev/ttyS0

       
dracut Warning: Could not boot.
dracut Warning: /dev/root does not exist

          
Generating "/run/initramfs/rdsosreport.txt"
-----------------------------------------------


Version-Release number of selected component (if applicable):
"RHEL-7.020130722.n.0 Server x86_64"


How reproducible: 100%


Steps to Reproduce:
1. In Beaker select system sgi-uv2-01.rhts.eng.bos.redhat.com
2. Then select  "RHEL-7.020130722.n.0 Server x86_64"
3. Look at the console to see the failure.

Actual results: Provisioning fails.


Expected results: Provisioning to work.


Additional info:  Fedora-19-Beta and Fedora19-Development fail the same way.
Specifying "root=/dev/sda1" on the linux boot line gets rid of "Warning: /dev/root does not exist" warning message, but it still complains "Could not boot." and stops in dracut.

Comment 1 Russ Anderson 2013-07-23 19:58:59 UTC
Update CC.

Comment 2 Russ Anderson 2013-07-23 20:00:40 UTC
Created attachment 777450 [details]
Additional failure details.

Comment 4 Harald Hoyer 2013-07-24 08:03:51 UTC
Well seems like a problem with the anaconda part. Don't know, maybe an kernel command line option for the network is missing...

Current kernel command line:

console=ttyS0,115200n8
debug
earlyprintk=ttyS0,115200n8
ks=http://beaker.engineering.redhat.com/kickstart/382267
ksdevice=08:00:69:15:C5:50
serial
netboot_method=efigrub

Comment 5 Will Woods 2013-07-24 19:39:12 UTC
Created attachment 777965 [details]
kickstart contents

Anaconda's dracut module will try to get a root device from the contents of the kickstart if the root device isn't otherwise specified.

I'm attaching the linked kickstart, just in case it goes away at some point.

Comment 6 Will Woods 2013-07-24 19:59:21 UTC
Anaconda's `parse-kickstart` command would generate the following boot arguments for the given kickstart: 

inst.text inst.repo=nfs:vtap-bos-eng01.bos.redhat.com:/vol/engineering/devarchive/redhat/nightly/RHEL-7.0-20130722.n.0/compose/Server/x86_64/os/

Normally that would be written to /etc/cmdline.d/80-kickstart.conf. That file doesn't seem to exist, which implies that the kickstart was never parsed.

So: dracut is bailing out before we have a chance to fetch/parse the kickstart.

I'm guessing that the "neednet" interface may have changed yet again, and anaconda isn't doing the right thing to get dracut to wait for the network.

Right now, anaconda does:

  set_neednet() {
      # if there's no netroot, make sure /tmp/net.ifaces exists
      [ -z "$netroot" ] && >> /tmp/net.ifaces
  }

to tell dracut to wait until the network comes up. Harald, is this the correct way to make sure dracut will wait for the network?

Comment 7 Harald Hoyer 2013-07-25 09:43:40 UTC
getargbool 0 rd.neednet && NEEDNET=1

# Don't continue if we don't need network
if [ -z "$netroot" ] && [ ! -e "/tmp/net.ifaces" ] && [ "$NEEDNET" != "1" ]; then
    return
fi

Comment 8 Harald Hoyer 2013-07-25 09:49:02 UTC
Can you rerun the installation with "rd.debug" added on the kernel command line?

Comment 9 Russ Anderson 2013-07-25 14:27:56 UTC
OK, trying it right now...

Comment 10 Russ Anderson 2013-07-25 14:53:24 UTC
Created attachment 778289 [details]
rdsosreport.txt with rd.debug

Failure output with rd.debug on the installer kernel boot line.

Comment 11 Harald Hoyer 2013-07-25 17:43:32 UTC
[   22.094830] dracut: ///lib/dracut/hooks/cmdline/26-parse-anaconda-kickstart.sh@8(source): kickstart=http://beaker.engineering.redhat.com/kickstart/386387
[   22.094920] dracut: ///lib/dracut/hooks/cmdline/26-parse-anaconda-kickstart.sh@9(source): '[' -z http://beaker.engineering.redhat.com/kickstart/386387 ']'
[   22.095010] dracut: ///lib/dracut/hooks/cmdline/26-parse-anaconda-kickstart.sh@13(source): '[' http://beaker.engineering.redhat.com/kickstart/386387 ']'
[   22.095086] dracut: ///lib/dracut/hooks/cmdline/26-parse-anaconda-kickstart.sh@13(source): '[' -z '' ']'
[   22.095161] dracut: ///lib/dracut/hooks/cmdline/26-parse-anaconda-kickstart.sh@13(source): root=anaconda-kickstart
[   22.095241] dracut: ///lib/dracut/hooks/cmdline/26-parse-anaconda-kickstart.sh@15(source): case "${kickstart%%:*}" in
[   22.095313] dracut: ///lib/dracut/hooks/cmdline/26-parse-anaconda-kickstart.sh@17(source): set_neednet
[   22.095376] dracut: //lib/anaconda-lib.sh@165(set_neednet): '[' -z '' ']'
[   22.095444] dracut: ///lib/dracut/hooks/cmdline/26-parse-anaconda-kickstart.sh@33(source): export kickstart

....

[   23.711280] dracut: ///lib/dracut/hooks/cmdline/28-parse-anaconda-net.sh@38(source): bootif=01-08-00-69-15-C5-50
[   23.724198] dracut: ///lib/dracut/hooks/cmdline/28-parse-anaconda-net.sh@39(source): warn ''''ksdevice=<MAC>''' is deprecated. Using BOOTIF=01-08-00-69-15-C5-50 instead.'
[   23.742707] dracut: //lib/dracut-lib.sh@411(warn): check_quiet
[   23.750746] dracut: //lib/dracut-lib.sh@397(check_quiet): '[' -z yes ']'
[   23.759798] dracut: //lib/dracut-lib.sh@412(warn): echo '<28>dracut Warning: '''ksdevice=<MAC>''' is deprecated. Using BOOTIF=01-08-00-69-15-C5-50 instead.'
[   23.776968] dracut: //lib/dracut-lib.sh@413(warn): echo 'dracut Warning: '''ksdevice=<MAC>''' is deprecated. Using BOOTIF=01-08-00-69-15-C5-50 instead.'
[   23.793737] dracut: dracut Warning: 'ksdevice=<MAC>' is deprecated. Using BOOTIF=01-08-00-69-15-C5-50 instead.
[   23.806448] dracut: ///lib/dracut/hooks/cmdline/28-parse-anaconda-net.sh@40(source): echo BOOTIF=01-08-00-69-15-C5-50
...

[   23.947078] dracut: ///lib/dracut/hooks/cmdline/28-parse-anaconda-net.sh@64(source): '[' -n 08:00:69:15:C5:50 ']'
[   23.960072] dracut: ///lib/dracut/hooks/cmdline/28-parse-anaconda-net.sh@64(source): set_neednet
[   23.971417] dracut: //lib/anaconda-lib.sh@165(set_neednet): '[' -z '' ']'
[   23.980549] dracut: ////lib/dracut/hooks/cmdline/28-parse-anaconda-net.sh@67(source): getarg inst.dhcpclass
....


[   29.118663] dracut: ///lib/dracut/hooks/cmdline/98-parse-ip-opts.sh@115(source): BOOTIF=01-08-00-69-15-C5-50
[   29.118742] dracut: ////lib/dracut/hooks/cmdline/98-parse-ip-opts.sh@116(source): fix_bootif 01-08-00-69-15-C5-50

...
[   33.606169] dracut: ////lib/dracut/hooks/pre-udev/60-net-genrules.sh@51(source): cat /tmp/net.bootdev
[   33.618006] dracut: ///lib/dracut/hooks/pre-udev/60-net-genrules.sh@51(source): bootdev=08:00:69:15:c5:50
...
[   34.049954] dracut: ///lib/dracut/hooks/pre-udev/60-net-genrules.sh@73(source): echo 'ATTR{address}=="08:00:69:15:c5:50", RUN+="/sbin/initqueue --onetime /sbin/ifup $env{INTERFACE} -m", GOTO="net_end"'

So, dracut installs ifup as an initqueue job for a network interface with MAC=="08:00:69:15:c5:50"

... Looks good to me... are you sure the MAC is correct?

Comment 12 Russ Anderson 2013-07-25 18:31:24 UTC
> ... Looks good to me... are you sure the MAC is correct?

Ah, the MAC was not correct.  The correct MAC is 08:00:69:16:8C:FE .

With the correct MAC the install gets further, but fails.  It created
BZ 988508 with details.

Comment 13 Russ Anderson 2013-07-25 21:09:54 UTC
Which Kickstart device MAC is correct?

"ksdevice=08:00:69:15:c5:50" hits this problem.
"ksdevice=08:00:69:16:8C:FE" hits BZ 988508 (probably because it cannot find the right kickstart device because that is the MAC of the UV2 system.

Comment 14 Russ Anderson 2013-08-06 22:33:51 UTC
What is the correct  Kickstart device MAC?

Still seeing the dracut "/dev/root does not exist" problem.

Comment 15 Siddhesh Poyarekar 2013-08-12 09:57:19 UTC
Sorry, I only meant to add myself to cc and did not notice that the option to clear needinfo is checked by default.  Setting needinfo again.

Comment 16 Will Woods 2013-08-13 18:38:39 UTC
(In reply to Russ Anderson from comment #13)
> Which Kickstart device MAC is correct?
> 
> "ksdevice=08:00:69:16:8C:FE" hits BZ 988508 (probably because it cannot find
> the right kickstart device because that is the MAC of the UV2 system.

No, I'm pretty sure bug 988508 is unrelated to networking. The logs there show that the install completed successfully, but there was a problem setting up the bootloader afterward.

As far as I can tell, there's no bug here - you just had the wrong MAC.

Comment 17 Russ Anderson 2013-09-18 12:29:48 UTC
Closing this as pilot error.  With the correct MAC address it is working.