Bug 1275956 - Broken upgrade / In Boot process show IOError info after upgrading RHEVH via TUI/AUTO/RHEVM, the original password is not available to login to new RHEV-H
Broken upgrade / In Boot process show IOError info after upgrading RHEVH via ...
Status: CLOSED ERRATA
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-node (Show other bugs)
3.6.0
Unspecified Unspecified
urgent Severity urgent
: ovirt-3.6.1
: 3.6.0
Assigned To: Ryan Barry
Huijuan Zhao
: Regression, TestBlocker
Depends On:
Blocks: 1199019 1206139
  Show dependency treegraph
 
Reported: 2015-10-28 04:07 EDT by yileye
Modified: 2016-03-09 09:41 EST (History)
13 users (show)

See Also:
Fixed In Version: ovirt-node-3.6.0-0.22.20151126git72acbb2.el7
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-03-09 09:41:43 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Node
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
/var/log/* + sosreport-log (5.26 MB, application/x-bzip)
2015-10-28 04:07 EDT, yileye
no flags Details
screenshot (1.46 MB, image/jpeg)
2015-10-28 04:09 EDT, yileye
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 48470 master MERGED recipe: Create a vdsm-like multipath.conf Never
oVirt gerrit 48679 ovirt-3.6 MERGED recipe: Create a vdsm-like multipath.conf Never

  None (edit)
Description yileye 2015-10-28 04:07:10 EDT
Created attachment 1087159 [details]
/var/log/* + sosreport-log

Description of problem:
In Boot process show "An error appeared in the UI:IOError"  when upgrade RHEV-H 7.1-20151015.0.el7ev to RHEV-H 7.2-20151025.0 via rhevm 3.5.5/TUI?AUTO

Version-Release number of selected component (if applicable):
RHEV-H 7.1-20151015.0.el7ev
ovirt-node-3.2.3-23.el7ev.noarch

RHEV-H 7.2-20151025.0.el7ev
ovirt-node-3.30-0.18.20151022git93dc63c.el7ev.noarch


How reproducible:
100%

Steps to Reproduce:
Step1.Clean TUI/AUTO install  RHEV-H 7.1-20151015.0.el7ev
Step2.Enable NIC via DHCP and enable SSH
Step3.Registered RHEVH to RHEVM 3.5.5-0.1.el6ev,
Step4.Upgrade it to RHEV-H 7.2-20151025.0.el7ev via RHEVM.

Actual results:
After Step4 ,we can see upgrade successful after auto reboot,but in boot process,it show “An error appeared in the UI:IOError(2,‘No such file or directory’)press ENTER to logout OR enter ‘s’ to drop to shell”,Press 'ENTER'
upgrade via TUI/AUTO also can reproduce this problem.
Then Press 'ENTER' , the original password is not available to login.

Expected results:
After Step4 ,there is no error in boot process.  
use the original password can login.

Additional info:
#ovirt-node.log:

<snip>
2015-10-28 00:54:35,102    WARNING Device is not disk /dev/sr0
2015-10-28 00:54:36,270      ERROR An error appeared in the UI: IOError(2, 'No such file or directory')
2015-10-28 00:54:36,270       INFO Exception:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ovirt/node/app.py", line 304, in run
    self.ui.run()
  File "/usr/lib/python2.7/site-packages/ovirt/node/ui/urwid_builder.py", line 437, in run
    self.navigate.to_first_plugin()
  File "/usr/lib/python2.7/site-packages/ovirt/node/ui/__init__.py", line 818, in to_first_plugin
    self.to_nth(0)
  File "/usr/lib/python2.7/site-packages/ovirt/node/ui/__init__.py", line 803, in to_nth
    self.to_plugin(plugin)
  File "/usr/lib/python2.7/site-packages/ovirt/node/ui/__init__.py", line 788, in to_plugin
    self.application.switch_to_plugin(plugin_candidate)
  File "/usr/lib/python2.7/site-packages/ovirt/node/app.py", line 243, in switch_to_plugin
    content = plugin.ui_content()
  File "/usr/lib/python2.7/site-packages/ovirt/node/installer/core/welcome_page.py", line 57, in ui_content
    ws += self.___installation_options()
  File "/usr/lib/python2.7/site-packages/ovirt/node/installer/core/welcome_page.py", line 93, in ___installation_options
    media = utils.system.InstallationMedia()
  File "/usr/lib/python2.7/site-packages/ovirt/node/utils/system.py", line 654, in __init__
    self.load()
  File "/usr/lib/python2.7/site-packages/ovirt/node/utils/system.py", line 658, in load
    data = get_media_version_number()
  File "/usr/lib/python2.7/site-packages/ovirtnode/ovirtfunctions.py", line 1539, in get_media_version_number
    upgrade_version = open("/live/syslinux/version")
IOError: [Errno 2] No such file or directory: '/live/syslinux/version'
2015-10-28 00:54:36,380    WARNING Device is already in dict: /dev/mapper/SAMSUNG_HD322GM_S2PAJ90B519911
2015-10-28 00:54:36,406    WARNING Type is rom

2015-10-28 00:54:36,406    WARNING Device is not disk /dev/sr0
2015-10-28 00:54:36,474    WARNING Device is already in dict: /dev/mapper/SAMSUNG_HD322GM_S2PAJ90B519911
2015-10-28 00:54:36,499    WARNING Type is rom
</snip>
Comment 1 yileye 2015-10-28 04:09 EDT
Created attachment 1087160 [details]
screenshot
Comment 2 Ying Cui 2015-10-28 04:44:11 EDT
This bug block about 50 test cases check for RHEV-H upgrade. Need to mark blocker and escalate it. Thanks.
Comment 3 Ryan Barry 2015-10-28 10:10:52 EDT
Note: this only blocks upgrades from 3.5 to 3.6, and should not block testing of upgrades to 3.5
Comment 4 Fabian Deutsch 2015-11-02 03:29:00 EST
In ovirt-node.log I see:

2015-10-28 00:54:32,066       INFO Starting '<module 'ovirt.node.installer' from '/usr/lib/python2.7/site-packages/ovirt/node/installer/__init__.pyc'>' application with '<class 'ovirt.node.ui.urwid_builder.UrwidUIBuilder'>' UI

Which indicates that the installer is getting started after step 4 (during the boot of the upgraded image), which is incorrect.
Comment 8 Fabian Deutsch 2015-11-02 08:03:24 EST
In addition to comment 4:

[    0.000000] Command line: BOOT_IMAGE=/vmlinuz0 root=live:LABEL=Root ro rootfstype=auto rootflags=ro ksdevice=bootif rd.dm=0 rd.md=0 crashkernel=256M lang= max_loop=256 rhgb quiet elevator=deadline rd.live.check rd.luks=0 rd.live.image

This shows that there is no kernel argument which should trigger the installer.
Comment 9 Ryan Barry 2015-11-02 09:39:13 EST
(In reply to Fabian Deutsch from comment #4)
> In ovirt-node.log I see:
> 
> 2015-10-28 00:54:32,066       INFO Starting '<module 'ovirt.node.installer'
> from '/usr/lib/python2.7/site-packages/ovirt/node/installer/__init__.pyc'>'
> application with '<class 'ovirt.node.ui.urwid_builder.UrwidUIBuilder'>' UI
> 
> Which indicates that the installer is getting started after step 4 (during
> the boot of the upgraded image), which is incorrect.

The installer is being started, because /etc/default/ovirt has the default data from the ISO in it.

The reason for this is that none of the logical volumes are detected, and persistence does not mount /config/etc/default/ovirt.
Comment 10 yileye 2015-11-04 07:47:20 EST
I am reproduce this bug on multipath FC,I will report tomorrow.
Comment 11 Fabian Deutsch 2015-11-04 10:04:19 EST
I can not reproduce it in dozens of tries in several VM configurations. Neitehr with multipaht, nor IDE, nor Virtio.
Comment 12 yileye 2015-11-05 02:47:59 EST
(In reply to Fabian Deutsch from comment #11)
> I can not reproduce it in dozens of tries in several VM configurations.
> Neitehr with multipaht, nor IDE, nor Virtio.

Hi Fabian,
I try to recurrence this bug on multipath FC/ISCSI,regrettably, have no  problem temporarily
but follow my description step,  I can  recurrence it 100% on single  path Disk machine.
Comment 13 Huijuan Zhao 2015-11-11 22:11:28 EST
Still encounter this issue on RHEV-H 7.2-20151104.0.

My steps:
1. TUI clean install RHEV-H 7.1-20151015.0.el7ev
2. Login RHEVH and change to Hosted Engine page
3. Setup additional host.
4. Upgrade it to RHEV-H 7.2-20151104.0.el7ev via TUI without change admin password successful.
5. Reboot RHEV-H.


Actual results:
After Step5 ,During boot RHEV-H, an error occured:

An error appeared in the UI:IOError(2,‘No such file or directory’)
press ENTER to logout OR enter ‘s’ to drop to shell”


Expected results:
After Step5 ,there is no error in boot process, Boot RHEV-H successful.
Comment 15 yileye 2015-11-12 07:08:57 EST
I can not reproduce this bug in rhev-hypervisor7-7.2-20151111.0.

Version-Release number of selected component (if applicable):
RHEV-H 7.1-20151015.0.el7ev
ovirt-node-3.2.3-23.el7ev.noarch

rhev-hypervisor7-7.2-20151111.0


Followed my description step :

Steps to Reproduce:

Step1.Clean TUI/AUTO install  RHEV-H 7.1-20151015.0.el7ev
Step2.Enable NIC via DHCP and enable SSH
Step3.Registered RHEVH to RHEVM 3.5.5-0.1.el6ev,
Step4.Upgrade it to rhev-hypervisor7-7.2-20151111.0 via RHEVM.

I had already reproduce this bug on single path ISCSI/Local Disk ,have no this bug.Single path FC machine need to lab to debug.So i will reproduce this bug on single path fC if it fixed.Delay to tomorrow to report status
Comment 16 Ying Cui 2015-11-12 10:44:13 EST
per IRC with fabian, I rephrase the comment 15 as following:

Leye tested this issue on build rhev-hypervisor7-7.2-20151111.0.

Local disk - PASS
singlepath iSCSI - PASS
singlepath FC - test in process.

Test steps:
Step1.Clean TUI/AUTO install  RHEV-H 7.1-20151015.0.el7ev
Step2.Enable NIC via DHCP and enable SSH
Step3.Registered RHEVH to RHEVM 3.5.5-0.1.el6ev,
Step4.Upgrade it to rhev-hypervisor7-7.2-20151111.0 via RHEVM.
Step5.After upgrade, can login new RHEV-H, no original bug issue.
Comment 17 Ying Cui 2015-11-26 03:22:34 EST
(In reply to yileye from comment #15)
> 
> I had already reproduce this bug on single path ISCSI/Local Disk ,have no
> this bug.Single path FC machine need to lab to debug.So i will reproduce
> this bug on single path fC if it fixed.Delay to tomorrow to report status

Leye, any update for singlepath FC you tested?
Comment 20 yileye 2015-11-30 10:38:26 EST
I tested this issue on build rhev-hypervisor7-7.2-20151111.0.

Version-Release number of selected component (if applicable):

RHEV-H 7.1-20151015.0.el7ev
rhev-hypervisor7-7.2-20151111.0


Test steps:
Step1.Clean TUI/AUTO install  RHEV-H 7.1-20151015.0.el7ev
Step2.Enable NIC via DHCP and enable SSH
Step3.Registered RHEVH to RHEVM 3.5
Step4.Upgrade it to rhev-hypervisor7-7.2-20151111.0 via RHEVM.
Step5.Auto reboot.

Local Disk - PASS
single path iSCSI - PASS
single path FC - PASS

test results:

After step5, After upgrade, can login new RHEV-H via correct usename and password, no original bug issue.
Comment 22 yileye 2015-12-09 05:00:18 EST
I tested this issue on build RHEV-H 7.2-20151201.2.el7ev.

Version-Release number of selected component (if applicable):

RHEV-H 7.1-20151015.0.el7ev
ovirt-node-3.2.3-23.el7ev.noarch

RHEV-H 7.2-20151201.2.el7ev
ovirt-node-3.6.0-0.23.20151201git5eed7af.el7ev.noarch


Test steps:
Step1.Clean TUI/AUTO install  RHEV-H 7.1-20151015.0.el7ev
Step2.Enable NIC via DHCP and enable SSH
Step3.Registered RHEVH to RHEVM 3.5
Step4.Upgrade it to rhev-hypervisor7-7.2-20151111.0 via RHEVM.
Step5.Auto reboot.

Local Disk - PASS
single path iSCSI - PASS
single path FC - PASS

test results:
After step5, After upgrade, can login new RHEV-H via correct usename and password, no original bug issue.

So the bug is fixed on ovirt-node-3.6.0-0.22.20151126git72acbb2.el7, change bug status to VERIFIED
Comment 26 errata-xmlrpc 2016-03-09 09:41:43 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0378.html

Note You need to log in before you can comment on or make changes to this bug.