Bug 1163688 - [3.5-7.0] rhevh7 - storage_init=ata - Automatic installation failed
Summary: [3.5-7.0] rhevh7 - storage_init=ata - Automatic installation failed
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-node
Version: 3.5.0
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: ---
: 3.5.0
Assignee: Ryan Barry
QA Contact: Virtualization Bugs
URL:
Whiteboard: node
: 1136392 (view as bug list)
Depends On:
Blocks: rhevh-7.0
TreeView+ depends on / blocked
 
Reported: 2014-11-13 09:52 UTC by Jiri Belka
Modified: 2016-02-10 20:02 UTC (History)
15 users (show)

Fixed In Version: rhev-hypervisor6-6.6-20141218.0.iso rhev-hypervisor7-7.0-20141218.0.iso
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-02-11 21:06:02 UTC
oVirt Team: Node
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
logs (45.09 KB, application/x-gzip)
2014-11-13 09:52 UTC, Jiri Belka
no flags Details
logs while using clean disk and storage_init=ata (44.19 KB, application/x-gzip)
2014-11-13 10:38 UTC, Jiri Belka
no flags Details
node.debug=1 logs (42.82 KB, application/x-gzip)
2014-11-14 11:59 UTC, Jiri Belka
no flags Details
/var/log/*.* (43.74 KB, application/x-gzip)
2014-12-16 03:11 UTC, cshao
no flags Details
tmp-ovirt.log (74.63 KB, text/plain)
2014-12-16 03:12 UTC, cshao
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2015:0160 0 normal SHIPPED_LIVE ovirt-node bug fix and enhancement update 2015-02-12 01:34:52 UTC
oVirt gerrit 35176 0 master MERGED Make sure $live_dev isn't an empty string Never
oVirt gerrit 35177 0 master MERGED Don't match cdroms on any bus Never
oVirt gerrit 35593 0 ovirt-3.5 MERGED Make sure $live_dev isn't an empty string Never
oVirt gerrit 35594 0 ovirt-3.5 MERGED Don't match cdroms on any bus Never
oVirt gerrit 36199 0 master MERGED Fix syntax in dracut lib Never
oVirt gerrit 36252 0 ovirt-3.5 MERGED Fix syntax in dracut lib Never

Description Jiri Belka 2014-11-13 09:52:56 UTC
Created attachment 957061 [details]
logs

Description of problem:

"reinstall" fails when there's already rhevh7 installed on the disk.

...
dell-r210ii-04 login: Automatic Installation
----------------------
Checking pre-conditions ...
(1/9) Prepare installation
(2/9) Performing automatic disk partitioning
HostVG/AppVG exists on a separate disk
Manual Intervention required

An error occurred while applying the changes:
Automatic installation failed. Please review /var/log/ovirt.log
'<type 'exceptions.RuntimeError'>' on transaction '<Transaction elements='[<PrepareInstallation 'Prepare installation'>, <AutomaticDiskPartitioning 'Performing automatic disk partitioning'>, <ConfigureNetworking 'Configuring network'>, <EnableSshPasswordAuthentication 'Ena
bling SSH password authentication'>, <ConfigureLogging 'Configuring Logging'>, <PerformInstallation 'Transferring image'>, <ConfigureKdump 'Configuring KDump'>, <InstallBootloader 'Installing Bootloader'>, <RunHooks 'Running Hooks'>]' title='Automatic Installation' at 0x4b
df6d0>': Automatic installation failed. Please review /var/log/ovirt.log - Automatic installation failed. Please review /var/log/ovirt.log
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ovirt/node/utils/console.py", line 119, in __run_transaction
  File "/usr/libexec/ovirt-auto-install", line 68, in commit
    "Please review /var/log/ovirt.log")
RuntimeError: Automatic installation failed. Please review /var/log/ovirt.log
Traceback (most recent call last):
  File "/usr/libexec/ovirt-auto-install", line 276, in <module>
    TransactionProgress(tx, is_dry=False).run()
  File "/usr/lib/python2.7/site-packages/ovirt/node/utils/console.py", line 94, in run
  File "/usr/lib/python2.7/site-packages/ovirt/node/utils/console.py", line 119, in __run_transaction
  File "/usr/libexec/ovirt-auto-install", line 68, in commit
    "Please review /var/log/ovirt.log")
RuntimeError: Automatic installation failed. Please review /var/log/ovirt.log
Nov 13 10:41:08 Automatic installation failed. Please review console messages.
Nov 13 10:41:08 Press Enter to drop to emergency shell.

[root@dell-r210ii-04 /]# grep ERROR /var/log/ovirt.log
2014-11-13 10:41:08,484 - ERROR - ovirtfunctions - There appears to already be an installation on another device:
2014-11-13 10:41:08,485 - ERROR - ovirtfunctions - /dev/mapper/WDC_WD2502ABYS-18B7A0_WD-WMAT16784517p4
2014-11-13 10:41:08,485 - ERROR - ovirtfunctions - The installation cannot proceed until the device is removed
2014-11-13 10:41:08,485 - ERROR - ovirtfunctions - from the system of the HostVG volume group is removed

[root@dell-r210ii-04 /]# cat /proc/cmdline 
rootflags=loop initrd=images/RHEVH/rhev-hypervisor7-7.0-20141107.0/initrd0.img root=live:/rhev-hypervisor7-7.0-20141107.0.iso rootfstype=auto ro rd.live.image rd.live.check rd.lvm=0 rd_NO_MULTIPATH rootflags=ro crashkernel=128M elevator=deadline install max_loop=256 rd.luks=0 rd.md=0 rd.dm=0 reinstall storage_init=ata storage_vol=::::: ssh_pwauth=1 adminpw=$1$a0gLYJ6j$N2cLwUuWx5o5dgVnD7M9L/ rhevm_admin_password=$1$a0gLYJ6j$N2cLwUuWx5o5dgVnD7M9L/ ntp=10.34.32.125 console=tty0 console=ttyS1,115200n8 BOOTIF=01-d0-67-e5-ee-65-09 BOOT_IMAGE=images/RHEVH/rhev-hypervisor7-7.0-20141107.0/vmlinuz0

^^^ 'reinstall' is used!

[root@dell-r210ii-04 /]# pvs
  PV                                                  VG     Fmt  Attr PSize   PFree
  /dev/mapper/WDC_WD2502ABYS-18B7A0_WD-WMAT16784517p4 HostVG lvm2 a--  232.11g    0 

[root@dell-r210ii-04 /]# multipath -ll
WDC_WD2502ABYS-18B7A0_WD-WMAT16784517 dm-0 ATA     ,WDC WD2502ABYS-1
size=233G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  `- 0:0:0:0 sda 8:0 active ready running

[root@dell-r210ii-04 /]# lvs
  LV      VG     Attr       LSize   Pool Origin Data%  Move Log Cpy%Sync Convert
  Config  HostVG -wi-ao----   8.00m                                             
  Data    HostVG -wi-a----- 226.11g                                             
  Logging HostVG -wi-a-----   2.00g                                             
  Swap    HostVG -wi-a-----   4.00g

[root@dell-r210ii-04 /]# dmesg | egrep sd[[:alpha:]]+
[    5.591825] sd 0:0:0:0: [sda] 488281250 512-byte logical blocks: (250 GB/232 GiB)
[    5.599543] sd 0:0:0:0: [sda] Write Protect is off
[    5.604461] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
[    5.604475] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    5.662494]  sda: sda1 sda2 sda3 sda4
[    5.666777] sd 0:0:0:0: [sda] Attached SCSI disk


Version-Release number of selected component (if applicable):
Red Hat Enterprise Virtualization Hypervisor release 7.0 (20141107.0.el7ev)

How reproducible:
100%

Steps to Reproduce:
1. clean disk, install rhevh7
2. when done, power reset, install again while using 'reinstall' as kernel param
3.

Actual results:
fails as storage has known lvm structure

Expected results:
should ignore existing data on the storage and reinstall rhevh

Additional info:

Comment 1 Jiri Belka 2014-11-13 09:59:41 UTC
this must be related to 'storage_init=ata' kernel param as i have no problem to reinstall the rhevh7 while using 'storage_init=/dev/sda'.

Comment 2 Jiri Belka 2014-11-13 10:27:27 UTC
i tried again with clean disk while using 'storage_init=ata' and it failed as well, thus it seems not to be related only to "dirty" disk.

Red Hat Enterprise Virtualization Hypervisor release 7.0 (20141107.0.el7ev)
Kernel 3.10.0-123.9.3.el7.x86_64 on an x86_64 (ttyS1)
dell-r210ii-04 login: Automatic Installation
----------------------
Checking pre-conditions ...
(1/9) Prepare installation
(2/9) Performing automatic disk partitioning

An error occurred while applying the changes:
invalid literal for int() with base 10: ''
'<type 'exceptions.ValueError'>' on transaction '<Transaction elements='[<PrepareInstallation 'Prepare installation'>, <AutomaticDiskPartitioning 'Performing automatic disk partitioning'>, <ConfigureNetworking 'Configuring network'>, <EnableSshPasswordAuthentication 'Enabling SSH password authentication'>, <ConfigureLogging 'Configuring Logging'>, <PerformInstallation 'Transferring image'>, <ConfigureKdump 'Configuring KDump'>, <InstallBootloader 'Installing Bootloader'>, <RunHooks 'Running Hooks'>]' title='Automatic Installation' at 0x33e9690>': invalid literal for int() with base 10: '' - invalid literal for int() with base 10: ''
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ovirt/node/utils/console.py", line 119, in __run_transaction
  File "/usr/libexec/ovirt-auto-install", line 56, in commit
    if storage_auto():
  File "/usr/lib/python2.7/site-packages/ovirtnode/storage.py", line 1023, in storage_auto
  File "/usr/lib/python2.7/site-packages/ovirtnode/storage.py", line 738, in perform_partitioning
  File "/usr/lib/python2.7/site-packages/ovirtnode/storage.py", line 928, in check_partition_sizes
  File "/usr/lib/python2.7/site-packages/ovirtnode/storage.py", line 184, in get_drive_size
ValueError: invalid literal for int() with base 10: ''
Traceback (most recent call last):
  File "/usr/libexec/ovirt-auto-install", line 276, in <module>
    TransactionProgress(tx, is_dry=False).run()
  File "/usr/lib/python2.7/site-packages/ovirt/node/utils/console.py", line 94, in run
  File "/usr/lib/python2.7/site-packages/ovirt/node/utils/console.py", line 119, in __run_transaction
  File "/usr/libexec/ovirt-auto-install", line 56, in commit
    if storage_auto():
  File "/usr/lib/python2.7/site-packages/ovirtnode/storage.py", line 1023, in storage_auto
  File "/usr/lib/python2.7/site-packages/ovirtnode/storage.py", line 738, in perform_partitioning
  File "/usr/lib/python2.7/site-packages/ovirtnode/storage.py", line 928, in check_partition_sizes
  File "/usr/lib/python2.7/site-packages/ovirtnode/storage.py", line 184, in get_drive_size
ValueError: invalid literal for int() with base 10: ''
Nov 13 11:24:41 Automatic installation failed. Please review console messages.
Nov 13 11:24:41 Press Enter to drop to emergency shell.
Nov 13 11:24:41 Reason: Transactio issue

Comment 3 Jiri Belka 2014-11-13 10:38:19 UTC
Created attachment 957113 [details]
logs while using clean disk and storage_init=ata

Comment 4 Jiri Belka 2014-11-13 10:44:38 UTC
*** Bug 1136392 has been marked as a duplicate of this bug. ***

Comment 5 Fabian Deutsch 2014-11-13 12:04:16 UTC
Can you please re-run this installation and add the node.debug=1 keyword to the commandline, and please provide /var/log and /tmp

Comment 6 Ryan Barry 2014-11-13 17:54:56 UTC
I'd also like the contents of /dev/disk/by-id/

I haven't been able to reproduce this, and as much information as possible would be helpful. Up to and including an sosreport.

Comment 7 Ryan Barry 2014-11-13 17:56:56 UTC
Incidentally, the contents of /etc/default/ovirt also.

Comment 8 Jiri Belka 2014-11-14 11:59:40 UTC
Created attachment 957557 [details]
node.debug=1 logs

i can't reproduce on my other server, so it is probably related to specific hw. sosreport doesn't work on emergency shell

Comment 9 Ryan Barry 2014-11-14 17:10:42 UTC
If sosreport doesn't work, please add whatever it spits out and potentially file a bug. sosreport may have changed under us. The emergency shell should be the same as the installed system.

Is this system PXE booted?

Comment 11 Ryan Barry 2014-11-14 17:54:39 UTC
There appear to have been two problems.

First is that evaluation of whether or not an empty string exists can now return true from interpolation, it seems.

a='' [ -e $a ] && echo true

This means that a failed grep for /dev/.initramfs/live won't pick up PXE booting, and will improperly fall through with an empty string for the live device.

The second is that the check for a storage device seems to have been bad.

Previously, it checked to make sure the device it was checking was not the live device (in this case, an empty string, so that check would succeed no matter what). Presumably under the assumption that a machine booted from CDROM would only have one CDROM device, and it would match.

This worked ok under EL6, since the cdrom appeared under the SCSI bus, and, under my very brief testing, it doesn't appear in /dev/disk/by-id/ anyway. It does in EL7. And it identifies as an ATA disk with udev.

So now we'll check to make sure it's not a CDROM, and that the variable holding the live device is not empty *as well* as the live device existing.

Comment 12 cshao 2014-11-17 03:17:31 UTC
Hi fabiand,

Seem they are the same issue with bug https://bugzilla.redhat.com/show_bug.cgi?id=1136392#c17

Comment 13 cshao 2014-11-17 03:24:43 UTC
(In reply to shaochen from comment #12)
> Hi fabiand,
> 
> Seem they are the same issue with bug
> https://bugzilla.redhat.com/show_bug.cgi?id=1136392#c17

OK, I noticed the old bug was closed by the new one due to this bug is more clear that it is related to storage_init=ata. So please ignore my comments :)

Comment 15 Jiri Belka 2014-11-18 12:59:38 UTC
i can't reproduce with rhev-hypervisor7-7.0-20141117.0, both as clean install and reinstall.

Comment 16 Fabian Deutsch 2014-11-18 13:43:13 UTC
Merged the patch according to comment 15

Comment 20 cshao 2014-12-16 03:10:12 UTC
Test version:
rhev-hypervisor7-7.0-20141212.0.iso
ovirt-node-3.1.0-0.34.20141210git0c9c493.el7.noarch

Test steps:
1. Auto install RHEV-H with below parameters:
# cat /proc/cmdline 
initrd=/images/rhevh-vdsm7-7.0-20141212.0/initrd0.img ksdevice=bootif rootflags=loop rootflags=ro rd.lvm=0 rd_NO_MULTIPATH rd.dm=0 elevator=deadline rootfstype=auto lang=  max_loop=256 quiet rd.md=0 rd.live.check rd.luks=0 install ro root=live:/rhev-hypervisor7-7.0-20141212.0.iso rd.live.image crashkernel=128M  BOOTIF=01-00-24-21-7f-b7-19 storage_init=ata adminpw=OKr05SbCu3D3g firstboot BOOT_IMAGE=/images/rhevh-vdsm7-7.0-20141212.0/vmlinuz0 


Test result:
Automatic installation still failed with "storage_init=ata" parameter.
So change bug status to ASSIGNED.

========================================================
Device specified in storage_init_init does not exist.
Other more log info please refer attachment.

Thanks!

Comment 21 cshao 2014-12-16 03:11:41 UTC
Created attachment 969396 [details]
/var/log/*.*

Comment 22 cshao 2014-12-16 03:12:53 UTC
Created attachment 969397 [details]
tmp-ovirt.log

Comment 23 cshao 2015-01-06 07:12:28 UTC
Test version:
rhev-hypervisor6-6.6-20150105.0
ovirt-node-3.1.0-0.39.20150105gitb784105.el6.noarch

rhev-hypervisor7-7.0-20150105.0
ovirt-node-3.1.0-0.39.20150105gitb784105.el7.noarch

Test steps:
1. Auto install RHEV-H with storage_init=ata parameters:

BOOT_IMAGE=/vmlinuz0 root=live:LABEL=Root ro rootfstype=auto rootflags=ro ksdevice=bootif rd.lvm=0 rd.dm=0 elevator=deadline lang= max_loop=256 quiet rd.md=0 rd.live.check rd.luks=0 rd.live.image crashkernel=128M BOOTIF=eth0 storage_init=ata adminpw=xxxxxx firstboot

Test result:
Auto install can succeed with "storage_init=ata" parameter.

So the bug is fixed, change bug status to VERIFIED.

Comment 25 errata-xmlrpc 2015-02-11 21:06:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2015-0160.html


Note You need to log in before you can comment on or make changes to this bug.