Bug 738630

Summary: Upgrade failed when / is on iSCSI
Product: Red Hat Enterprise Linux 6 Reporter: Alexander Todorov <atodorov>
Component: anacondaAssignee: Ales Kozumplik <akozumpl>
Status: CLOSED NOTABUG QA Contact: Release Test Team <release-test-team>
Severity: high Docs Contact:
Priority: high    
Version: 6.2CC: jzeleny
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-09-19 09:29:38 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 691780    

Description Alexander Todorov 2011-09-15 12:30:50 UTC 
Description of problem:

Upgrade from 6.1 to 6.2 failed. The system wasn't able to boot post upgrade. 

Version-Release number of selected component (if applicable):
anaconda-13.21.138-1.el6.x86_64

How reproducible:


Steps to Reproduce (kvm guest):
1. Install RHEL 6.1 - default install. 
2. Use one local disk and one iSCSI disk. Use default partitioning (where the iSCSI disk will be marked as a PV which is part of the VG which holds /)
3. Perform an upgrade. Add the same iSCSI disk before anaconda scans all disks
4. Perform the upgrade and select to update boot loader.
  
Actual results:
Upgrade finishes and system reboots. It fails to boot. 

Expected results:
System find / and boots successfully. 

Additional info:

I think eth0 changed name to eth1 but I'm not sure. In another upgrade (where / was on local disk eth0 was renamed). Will boot into rescue mode and fetch logs.


Console log:

Performance Events: Broken PMU hardware detected, using software events only.
NMI watchdog disabled (cpu0): hardware events not enabled
Brought up 1 CPUs
Total of 1 processors activated (5332.88 BogoMIPS).
devtmpfs: initialized
regulator: core version 0.5
NET: Registered protocol family 16
ACPI: bus type pci registered
PCI: Using configuration type 1 for base access
bio: create slab <bio-0> at 0
ACPI: Interpreter enabled
ACPI: (supports S0 S3 S4 S5)
ACPI: Using IOAPIC for interrupt routing
ACPI: No dock devices found.
HEST: Table not found.
ACPI: PCI Root Bridge [PCI0] (0000:00)
pci 0000:00:01.3: quirk: region b000-b03f claimed by PIIX4 ACPI
pci 0000:00:01.3: quirk: region b100-b10f claimed by PIIX4 SMB
ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11)
ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11)
ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11)
ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11)
ACPI: PCI Interrupt Link [LNKS] (IRQs 9) *0
vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none
vgaarb: loaded
vgaarb: bridge control possible 0000:00:02.0
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
PCI: Using ACPI for IRQ routing
NetLabel: Initializing
NetLabel:  domain hash size = 128
NetLabel:  protocols = UNLABELED CIPSOv4
NetLabel:  unlabeled traffic allowed by default
Switching to clocksource kvm-clock
pnp: PnP ACPI init
ACPI: bus type pnp registered
pnp: PnP ACPI: found 7 devices
ACPI: ACPI bus type pnp unregistered
NET: Registered protocol family 2
IP route cache hash table entries: 32768 (order: 6, 262144 bytes)
TCP established hash table entries: 131072 (order: 9, 2097152 bytes)
TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
TCP: Hash tables configured (established 131072 bind 65536)
TCP reno registered
NET: Registered protocol family 1
pci 0000:00:00.0: Limiting direct PCI/PCI transfers
pci 0000:00:01.0: PIIX3: Enabling Passive Release
pci 0000:00:01.0: Activating ISA DMA hang workarounds
Trying to unpack rootfs image as initramfs...
Freeing initrd memory: 23478k freed
audit: initializing netlink socket (disabled)
type=2000 audit(1316089412.671:1): initialized
HugeTLB registered 2 MB page size, pre-allocated 0 pages
VFS: Disk quotas dquot_6.5.2
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
msgmni has been set to 1985
alg: No test for stdrng (krng)
ksign: Installing public key data
Loading keyring
- Added public key DE7DE479116DE38
- User ID: Red Hat, Inc. (Kernel Module GPG key)
- Added public key D4A26C9CCD09BEDA
- User ID: Red Hat Enterprise Linux Driver Update Program <secalert>
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
pciehp: PCI Express Hot Plug Controller Driver version: 0.4
acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
acpiphp: Slot [1] registered
acpiphp: Slot [2] registered
acpiphp: Slot [3] registered
acpiphp: Slot [4] registered
acpiphp: Slot [5] registered
acpiphp: Slot [6] registered
acpiphp: Slot [7] registered
acpiphp: Slot [8] registered
acpiphp: Slot [9] registered
acpiphp: Slot [10] registered
acpiphp: Slot [11] registered
acpiphp: Slot [12] registered
acpiphp: Slot [13] registered
acpiphp: Slot [14] registered
acpiphp: Slot [15] registered
acpiphp: Slot [16] registered
acpiphp: Slot [17] registered
acpiphp: Slot [18] registered
acpiphp: Slot [19] registered
acpiphp: Slot [20] registered
acpiphp: Slot [21] registered
acpiphp: Slot [22] registered
acpiphp: Slot [23] registered
acpiphp: Slot [24] registered
acpiphp: Slot [25] registered
acpiphp: Slot [26] registered
acpiphp: Slot [27] registered
acpiphp: Slot [28] registered
acpiphp: Slot [29] registered
acpiphp: Slot [30] registered
acpiphp: Slot [31] registered
input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
ACPI: Power Button [PWRF]
processor LNXCPU:00: registered as cooling_device0
ERST: Table is not found!
GHES: HEST is not enabled!
hpet_acpi_add: no address or irqs in _CRS
Non-volatile memory driver v1.3
Linux agpgart interface v0.103
crash memory driver: version 1.1
Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
�serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
00:05: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
brd: module loaded
loop: module loaded
input: Macintosh mouse button emulation as /devices/virtual/input/input1
Fixed MDIO Bus: probed
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
uhci_hcd: USB Universal Host Controller Interface driver
ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11
uhci_hcd 0000:00:01.2: PCI INT D -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11
uhci_hcd 0000:00:01.2: UHCI Host Controller
uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1
uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c020
usb usb1: New USB device found, idVendor=1d6b, idProduct=0001
usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb1: Product: UHCI Host Controller
usb usb1: Manufacturer: Linux 2.6.32-195.el6.x86_64 uhci_hcd
usb usb1: SerialNumber: 0000:00:01.2
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 2 ports detected
PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
serio: i8042 KBD port at 0x60,0x64 irq 1
serio: i8042 AUX port at 0x60,0x64 irq 12
mice: PS/2 mouse device common for all mice
input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input2
rtc_cmos 00:01: RTC can wake from S4
rtc_cmos 00:01: rtc core: registered rtc_cmos as rtc0
rtc0: alarms up to one day, 114 bytes nvram
cpuidle: using governor ladder
cpuidle: using governor menu
usbcore: registered new interface driver hiddev
usbcore: registered new interface driver usbhid
usbhid: v2.6:USB HID core driver
TCP cubic registered
Initializing XFRM netlink socket
NET: Registered protocol family 17
registered taskstats version 1
rtc_cmos 00:01: setting system clock to 2011-09-15 12:23:31 UTC (1316089411)
Initalizing network drop monitor service
Freeing unused kernel memory: 1244k freed
Write protecting the kernel read-only data: 10240k
Freeing unused kernel memory: 1044k freed
Freeing unused kernel memory: 1764k freed
dracut: dracut-004-234.el6
dracut: rd_NO_LUKS: removing cryptoluks activation
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
cnic: Broadcom NetXtreme II CNIC Driver cnic v2.5.3 (June 6, 2011)
Loading iSCSI transport class v2.0-870.
Broadcom NetXtreme II iSCSI Driver bnx2i v2.7.0.3 (Jun 15, 2010)
iscsi: registered transport (bnx2i)
device-mapper: uevent: version 1.0.3
device-mapper: ioctl: 4.21.6-ioctl (2011-07-06) initialised: dm-devel
udev: starting version 147
usb 1-1: new full speed USB device using uhci_hcd and address 2
dracut: Starting plymouth daemon
input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input3
dracut: rd_NO_DM: removing DM RAID activation
dracut: rd_NO_MD: removing MD RAID activation
usb 1-1: New USB device found, idVendor=0627, idProduct=0001
usb 1-1: New USB device strings: Mfr=1, Product=3, SerialNumber=5
usb 1-1: Product: QEMU USB Tablet
usb 1-1: Manufacturer: QEMU 0.12.1
usb 1-1: SerialNumber: 42
usb 1-1: configuration #1 chosen from 1 choice
input: QEMU 0.12.1 QEMU USB Tablet as /devices/pci0000:00/0000:00:01.2/usb1/1-1/1-1:1.0/input/input4
generic-usb 0003:0627:0001.0001: input,hidraw0: USB HID v0.01 Pointer [QEMU 0.12.1 QEMU USB Tablet] on usb-0000:00:01.2-1/input0
scsi0 : ata_piix
scsi1 : ata_piix
ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14
ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15
ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 10
virtio-pci 0000:00:03.0: PCI INT A -> Link[LNKC] -> GSI 10 (level, high) -> IRQ 10
ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10
virtio-pci 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI 10 (level, high) -> IRQ 10
ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 11
virtio-pci 0000:00:06.0: PCI INT A -> Link[LNKB] -> GSI 11 (level, high) -> IRQ 11
udev: renamed network interface eth0 to eth0-renamed
 vda: vda1 vda2
dracut: Scanning devices vda2  for LVM logical volumes vg_dhcp70249/lv_root vg_dhcp70249/lv_swap 
dracut: Couldn't find device with uuid Y3xLrz-TU4f-mAl2-Du0s-3ijY-pHIX-BWbZkd.
dracut: inactive '/dev/vg_dhcp70249/lv_root' [25.54 GiB] inherit
dracut: inactive '/dev/vg_dhcp70249/lv_swap' [1.97 GiB] inherit
dracut: Couldn't find device with uuid Y3xLrz-TU4f-mAl2-Du0s-3ijY-pHIX-BWbZkd.
dracut: Refusing activation of partial LV lv_root. Use --partial to override.
dracut: Scanning devices vda2  for LVM logical volumes vg_dhcp70249/lv_root vg_dhcp70249/lv_swap 
dracut: Couldn't find device with uuid Y3xLrz-TU4f-mAl2-Du0s-3ijY-pHIX-BWbZkd.
dracut: inactive '/dev/vg_dhcp70249/lv_root' [25.54 GiB] inherit
dracut: ACTIVE '/dev/vg_dhcp70249/lv_swap' [1.97 GiB] inherit
dracut: Partial mode. Incomplete logical volumes will be processed.
dracut: Couldn't find device with uuid Y3xLrz-TU4f-mAl2-Du0s-3ijY-pHIX-BWbZkd.
Buffer I/O error on device dm-2, logical block 0
Buffer I/O error on device dm-2, logical block 0
Buffer I/O error on device dm-2, logical block 1
Buffer I/O error on device dm-2, logical block 1
Buffer I/O error on device dm-2, logical block 1
Buffer I/O error on device dm-2, logical block 1
Buffer I/O error on device dm-2, logical block 0
Buffer I/O error on device dm-2, logical block 0
Buffer I/O error on device dm-2, logical block 0
Buffer I/O error on device dm-2, logical block 0
mount: you must specify the filesystem type
mount: you must specify the filesystem type
dracut Warning: Can't mount root filesystem


mount: you must specify the filesystem type
mount: you must specify the filesystem type
mount: you must specify the filesystem type
mount: you must specify the filesystem type
mount: you must specify the filesystem type
mount: you must specify the filesystem type
mount: you must specify the filesystem type
mount: you must specify the filesystem type
mount: you must specify the filesystem type
mount: you must specify the filesystem type
mount: you must specify the filesystem type
mount: you must specify the filesystem type
mount: you must specify the filesystem type
mount: you must specify the filesystem type
mount: you must specify the filesystem type
mount: you must specify the filesystem type
mount: you must specify the filesystem type
mount: you must specify the filesystem type
mount: you must specify the filesystem type
mount: you must specify the filesystem type
mount: you must specify the filesystem type



dracut Warning: Boot has failed. To debug this issue add "rdshell" to the kernel command line.


dracut Warning: Signal caught!

dracut Warning: Boot has failed. To debug this issue add "rdshell" to the kernel command line.
Kernel panic - not syncing: Attempted to kill init!
Pid: 1, comm: init Not tainted 2.6.32-195.el6.x86_64 #1
Call Trace:
 [<ffffffff814eb71b>] ? panic+0x78/0x143
 [<ffffffff8106ee72>] ? do_exit+0x852/0x860
 [<ffffffff81177ae5>] ? fput+0x25/0x30
 [<ffffffff8106eed8>] ? do_group_exit+0x58/0xd0
 [<ffffffff8106ef67>] ? sys_exit_group+0x17/0x20
 [<ffffffff8100b0b2>] ? system_call_fastpath+0x16/0x1b
Comment 1 Alexander Todorov 2011-09-15 12:41:06 UTC 
Hmm, I found something which is probably my fault but I haven't seen it with older releases IIRC.

After the initial install I created another VM which has different MAC address for the nic and then upgraded. ifcfg-eth0 still contained the MAC from the previous VM. This is probably propagated to initrd but I can't find where.
Comment 2 Alexander Todorov 2011-09-15 12:46:10 UTC 
ifname=eth0:XXXX on the grub command line also contains the old MAC.

After fixing grub and ifcfg-eth0 to contain the correct MAC I hit bug #737479.
Comment 3 Ales Kozumplik 2011-09-19 07:53:09 UTC 
(In reply to comment #1)
> Hmm, I found something which is probably my fault but I haven't seen it with
> older releases IIRC.
> 
> After the initial install I created another VM which has different MAC address
> for the nic and then upgraded. ifcfg-eth0 still contained the MAC from the
> previous VM. This is probably propagated to initrd but I can't find where.

Alex can you retest under the standard conditions then?
Comment 4 Alexander Todorov 2011-09-19 09:29:38 UTC 
(In reply to comment #3)

> Alex can you retest under the standard conditions then?

Re-tested and I hit bug #737479.