Bug 1176058 - [3.5-7.0] Reinstall rhevh6.6/7.0 hosts from the rhevm 3.5 Administration Portal didn't work
Summary: [3.5-7.0] Reinstall rhevh6.6/7.0 hosts from the rhevm 3.5 Administration Port...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-node
Version: 3.5.0
Hardware: Unspecified
OS: Unspecified
high
urgent
Target Milestone: ---
: 3.5.0
Assignee: Douglas Schilling Landgraf
QA Contact: Virtualization Bugs
URL:
Whiteboard: node
Depends On:
Blocks: rhev35rcblocker rhev35gablocker
TreeView+ depends on / blocked
 
Reported: 2014-12-19 10:15 UTC by haiyang,dong
Modified: 2016-06-14 10:11 UTC (History)
19 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-12-29 15:08:30 UTC
oVirt Team: Node


Attachments (Terms of Use)
attached engine.log from rhevm 3.5 Administration Portal host (135.58 KB, text/plain)
2014-12-22 12:18 UTC, haiyang,dong
no flags Details
attached logs for rhev-hypervisor7-7.0-20141218.0.el7ev iso (4.94 MB, application/x-xz)
2014-12-22 12:20 UTC, haiyang,dong
no flags Details
attached logs for host-deploy dir (530.00 KB, application/x-tar)
2014-12-24 02:43 UTC, haiyang,dong
no flags Details
attached Screenshot for admin portal.png (25.69 KB, image/png)
2014-12-24 03:07 UTC, haiyang,dong
no flags Details
attached Screenshot for host windows.png (62.19 KB, image/png)
2014-12-24 08:47 UTC, haiyang,dong
no flags Details


Links
System ID Priority Status Summary Last Updated
oVirt gerrit 36301 master MERGED installer: Only add mapth.wwid once Never
oVirt gerrit 36561 ovirt-3.5 MERGED installer: Only add mapth.wwid once Never
oVirt gerrit 36575 master MERGED installer: Harden wwid detection Never
oVirt gerrit 36576 ovirt-3.5 MERGED installer: Harden wwid detection Never
Red Hat Bugzilla 1176048 None None None Never

Internal Links: 1176048

Description haiyang,dong 2014-12-19 10:15:47 UTC
Description of problem:
Reinstall rhevh7.0 hosts from the rhevm 3.5 Administration Portal actually didn't work 
in rhevm version 3.5.0-0.26.el6ev(vt13.3).It didn't reinstall rhevh7.0 hosts and restarting the physical server.

If someone want to know details info about Reinstalling_Virtualization_Hosts,please
see https://10.66.110.5/ovirt-engine/docs/manual/en-US/html/Administration_Guide/sect-Host_Tasks.html#Reinstalling_Virtualization_Hosts.

Version-Release number of selected component (if applicable):
rhev-hypervisor7-7.0-20141218.0.el7ev
ovirt-node-3.1.0-0.37.20141218gitcf277e1.el7.noarch
Red Hat Enterprise Virtualization Manager Version: 3.5.0-0.26.el6ev

How reproducible:
100%

Steps to Reproduce:
1. Install rhev-hypervisor7-7.0-20141218.0.el7ev into two hosts
2. Register these two rhevh 7.0 into RHEVM with the same Data Center 3.5 and cluster 3.5.
(Ensure that the cluster contains more than one host before performing a reinstall)
3. Approve them up.
4. Maintenance one host.
5. Click Reinstall to open the Install Host window.
6. Click OK to reinstall the host.
   
Actual results:
Alought the host displays a status of Up in fianlly, but
Reinstall rhevh7.0 hosts from the Administration Portal didn't work.
It didn't reinstall rhevh7.0 hosts and restarting the physical server.

Expected results:
Reinstall rhevh7.0 hosts from the Administration Portal works well.

Additional info:
..

Comment 1 Fabian Deutsch 2014-12-19 12:33:23 UTC
The issue here seems to be that the mpath.wwid=<wwid> karg is added to the cmdline twice. Removing the duplicate results in a clean boot.

Comment 2 haiyang,dong 2014-12-22 12:18:11 UTC
Created attachment 971985 [details]
attached engine.log from rhevm 3.5 Administration Portal host

Comment 3 haiyang,dong 2014-12-22 12:20:38 UTC
Created attachment 971986 [details]
attached logs for rhev-hypervisor7-7.0-20141218.0.el7ev iso

Comment 4 Fabian Deutsch 2014-12-22 13:11:40 UTC
The fix does not completely solve this issue.

Comment 5 Douglas Schilling Landgraf 2014-12-23 13:57:38 UTC
Hi hadong,

(In reply to haiyang,dong from comment #0)
> Description of problem:
> Reinstall rhevh7.0 hosts from the rhevm 3.5 Administration Portal actually
> didn't work 
> in rhevm version 3.5.0-0.26.el6ev(vt13.3).It didn't reinstall rhevh7.0 hosts
> and restarting the physical server.
> 
> If someone want to know details info about
> Reinstalling_Virtualization_Hosts,please
> see
> https://10.66.110.5/ovirt-engine/docs/manual/en-US/html/Administration_Guide/
> sect-Host_Tasks.html#Reinstalling_Virtualization_Hosts.
> 
> Version-Release number of selected component (if applicable):
> rhev-hypervisor7-7.0-20141218.0.el7ev
> ovirt-node-3.1.0-0.37.20141218gitcf277e1.el7.noarch
> Red Hat Enterprise Virtualization Manager Version: 3.5.0-0.26.el6ev
> 
> How reproducible:
> 100%
> 
> Steps to Reproduce:
> 1. Install rhev-hypervisor7-7.0-20141218.0.el7ev into two hosts
> 2. Register these two rhevh 7.0 into RHEVM with the same Data Center 3.5 and
> cluster 3.5.
> (Ensure that the cluster contains more than one host before performing a
> reinstall)
> 3. Approve them up.
> 4. Maintenance one host.
> 5. Click Reinstall to open the Install Host window.
> 6. Click OK to reinstall the host.
>    
> Actual results:
> Alought the host displays a status of Up in fianlly, but
> Reinstall rhevh7.0 hosts from the Administration Portal didn't work.
> It didn't reinstall rhevh7.0 hosts and restarting the physical server.

Could you please clarify what error are you facing?
Do you see any error on admin portal or in the node itself?

Additionaly to that, could you please share the logs from /var/log/ovirt-engine/host-deploy in rhevm machine?

Thanks!

Comment 6 haiyang,dong 2014-12-24 02:35:19 UTC
(In reply to Douglas Schilling Landgraf from comment #5)
> Hi hadong,
> 
> Could you please clarify what error are you facing?
> Do you see any error on admin portal or in the node itself?
I didn't see any error info from admin portal or node.
But reinstall rhevh7.0 hosts from the Administration Portal didn't work.
It didn't reinstall rhevh7.0 hosts and restarting the physical server.

About how to reinstall Virtualization Hosts, please review https://10.66.110.5/ovirt-engine/docs/manual/en-US/html/Administration_Guide/sect-Host_Tasks.html#Reinstalling_Virtualization_Hosts

> 
> Additionaly to that, could you please share the logs from
> /var/log/ovirt-engine/host-deploy in rhevm machine?
> 
> Thanks!

Comment 7 haiyang,dong 2014-12-24 02:43:37 UTC
Created attachment 972616 [details]
attached logs for host-deploy dir

Comment 8 haiyang,dong 2014-12-24 03:07:30 UTC
Created attachment 972620 [details]
attached Screenshot for admin portal.png

From attached Screenshot for admin portal.png, i noticed that there two buttons "Reinstall" and "Upgrade" in admin portal after Maintenance host.
What i guess is the follow:
"Reinstall" button functions is erase old version and install a new version
"Upgrade" button functions is keep old version and aslo install a new version

This bug was just only talking about "Reinstall" button functions seems didn't work in vt13.3, Not involved "Upgrade" button functions

Comment 9 Doron Fediuck 2014-12-24 07:55:52 UTC
(In reply to haiyang,dong from comment #8)
> Created attachment 972620 [details]
> attached Screenshot for admin portal.png
> 
> From attached Screenshot for admin portal.png, i noticed that there two
> buttons "Reinstall" and "Upgrade" in admin portal after Maintenance host.
> What i guess is the follow:
> "Reinstall" button functions is erase old version and install a new version
> "Upgrade" button functions is keep old version and aslo install a new version
> 
> This bug was just only talking about "Reinstall" button functions seems
> didn't work in vt13.3, Not involved "Upgrade" button functions

Hadong,
for rhev-h, both buttons will perform and upgrade action based on the selected
iso.

The logs you provided have no error. so the question is if this is still relevant
based on comment 5?

Comment 10 haiyang,dong 2014-12-24 08:47:47 UTC
Created attachment 972715 [details]
attached Screenshot for host windows.png

(In reply to Doron Fediuck from comment #9)
> (In reply to haiyang,dong from comment #8)
> 
> Hadong,
> for rhev-h, both buttons will perform and upgrade action based on the
> selected
> iso.
> 
> The logs you provided have no error. so the question is if this is still
> relevant
> based on comment 5?

If both buttons do the same action(upgrade), so the current question was for Install host windows(seen Host window.png), there wasn't Current version and could selected ISO name item were shown in Install host windows. The user could selected the iso what wanted upgraded version in Upgrade install windows.

Comment 11 Doron Fediuck 2014-12-24 11:06:43 UTC
(In reply to haiyang,dong from comment #10)
> Created attachment 972715 [details]
> attached Screenshot for host windows.png
 
> If both buttons do the same action(upgrade), so the current question was for
> Install host windows(seen Host window.png), there wasn't Current version and
> could selected ISO name item were shown in Install host windows. The user
> could selected the iso what wanted upgraded version in Upgrade install
> windows.

The same action is done from the engine. In the UI the reinstall used the
current ISO and upgrade will allow you to choose another one. Eventually
the same set of tasks will be executed, and this is why I said it's the same.

What is still missing for this BZ resolution ?

Comment 12 haiyang,dong 2014-12-25 03:00:51 UTC
OK, It's clear for me about "Reinstall" button functions now
Current the issue is rhev-hypervisor7-7.0-20141218.0.el7ev reinstall via "Reinstall" button in rhevm portal failed.
(a) In rhevm portal:
host will be up after clicking "Reinstall" button

Events message from rhevm portal
=====	
2014-Dec-25, 10:45
Status of host dhcp-10-114.nay.redhat.com was set to Up.
oVirt

2014-Dec-25, 10:45
Host dhcp-10-114.nay.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued.
oVirt

2014-Dec-25, 10:40
Status of host dhcp-10-114.nay.redhat.com was set to Up.
oVirt

2014-Dec-25, 10:40
Host dhcp-10-114.nay.redhat.com installed
5f0c3f4a
oVirt

2014-Dec-25, 10:40
Installing Host dhcp-10-114.nay.redhat.com. Stage: Termination.
5f0c3f4a
oVirt

2014-Dec-25, 10:40
Installing Host dhcp-10-114.nay.redhat.com. Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20141224214001-10.66.10.114-5f0c3f4a.log'.
5f0c3f4a
oVirt

2014-Dec-25, 10:40
Installing Host dhcp-10-114.nay.redhat.com. Stage: Pre-termination.
5f0c3f4a
oVirt

2014-Dec-25, 10:39
Installing Host dhcp-10-114.nay.redhat.com. Starting vdsm.
5f0c3f4a
oVirt

2014-Dec-25, 10:39
Installing Host dhcp-10-114.nay.redhat.com. Stopping libvirtd.
5f0c3f4a
oVirt

2014-Dec-25, 10:39
Installing Host dhcp-10-114.nay.redhat.com. Stage: Closing up.
5f0c3f4a
oVirt

2014-Dec-25, 10:39
Installing Host dhcp-10-114.nay.redhat.com. Stage: Transaction commit.
5f0c3f4a
oVirt

2014-Dec-25, 10:39
Installing Host dhcp-10-114.nay.redhat.com. Enrolling certificate.
5f0c3f4a
oVirt

2014-Dec-25, 10:39
Installing Host dhcp-10-114.nay.redhat.com. Setting time.
5f0c3f4a
oVirt

2014-Dec-25, 10:39
Installing Host dhcp-10-114.nay.redhat.com. Stage: Misc configuration.
5f0c3f4a
oVirt

2014-Dec-25, 10:39
Installing Host dhcp-10-114.nay.redhat.com. Stage: Package installation.
5f0c3f4a
oVirt

2014-Dec-25, 10:39
Installing Host dhcp-10-114.nay.redhat.com. Stage: Misc configuration.
5f0c3f4a
oVirt

2014-Dec-25, 10:39
Installing Host dhcp-10-114.nay.redhat.com. Stage: Transaction setup.
5f0c3f4a
oVirt

2014-Dec-25, 10:39
Installing Host dhcp-10-114.nay.redhat.com. Hardware supports virtualization.
5f0c3f4a
oVirt

2014-Dec-25, 10:39
Installing Host dhcp-10-114.nay.redhat.com. Stage: Setup validation.
5f0c3f4a
oVirt

2014-Dec-25, 10:39
Installing Host dhcp-10-114.nay.redhat.com. Disabling Kdump integration.
5f0c3f4a
oVirt

2014-Dec-25, 10:39
Installing Host dhcp-10-114.nay.redhat.com. Logs at host located at: '/tmp/ovirt-host-deploy-20141225023945-gknyso.log'.
5f0c3f4a
oVirt

2014-Dec-25, 10:39
Installing Host dhcp-10-114.nay.redhat.com. Host is hypervisor.
5f0c3f4a
oVirt

2014-Dec-25, 10:39
Installing Host dhcp-10-114.nay.redhat.com. Kdump unsupported.
5f0c3f4a
oVirt

2014-Dec-25, 10:39
Installing Host dhcp-10-114.nay.redhat.com. Stage: Environment customization.
5f0c3f4a
oVirt

2014-Dec-25, 10:39
Installing Host dhcp-10-114.nay.redhat.com. Stage: Programs detection.
5f0c3f4a
oVirt

2014-Dec-25, 10:39
Installing Host dhcp-10-114.nay.redhat.com. Stage: Environment packages setup.
5f0c3f4a
oVirt

2014-Dec-25, 10:39
Installing Host dhcp-10-114.nay.redhat.com. Stage: Environment setup.
5f0c3f4a
oVirt

2014-Dec-25, 10:39
Installing Host dhcp-10-114.nay.redhat.com. Stage: Initializing.
5f0c3f4a
oVirt

2014-Dec-25, 10:39
Installing Host dhcp-10-114.nay.redhat.com. Connected to host 10.66.10.114 with SSH key fingerprint: 33:7f:93:f3:fe:10:4f:43:5c:f2:40:3f:b9:51:0e:9b.
5f0c3f4a
oVirt

2014-Dec-25, 10:39
Host dhcp-10-114.nay.redhat.com configuration was updated by admin.
5f0c3f4a
oVirt

2014-Dec-25, 10:37
Host dhcp-10-114.nay.redhat.com was switched to Maintenance mode by admin.
524582d9
oVirt

(b) In rhevh 7.0 side:

The rhevh did not reboot automatically after clicking "Reinstall" button from rhevm portal. Also i didn't see any upgrade log info from ovirt.log/ovirt-node.log and didn't find ovirt-node-upgrade.log file from rhevh host.

After manually reboot rhevh, only one version should be selected, if it upgrade
success, there should be two version could be selected in boot grub

My test steps:
1. Install rhev-hypervisor7-7.0-20141218.0.el7ev successful.
2. Register rhev-hypervisor7-7.0-20141218.0.el7ev to rhevm successful.
3. Go to rhevm portal, approve it up.
4. Maintenance the host.
5. Click Reinstall button from rhevm portal.

Comment 13 Doron Fediuck 2014-12-25 12:38:35 UTC
Haiyang,
Re-install is installing the same OS version again.
So this is working as expected. ie- the same version should be seen
and no other ISO should be available. Using the same iso may not require
a reboot as well.

For upgrade scenario I'd expect an additional ISO, but this is not
what you describe. 

Can you please explain where and what is the problem?

Comment 14 haiyang,dong 2014-12-25 13:19:58 UTC
(In reply to Doron Fediuck from comment #13)
> Haiyang,
> Re-install is installing the same OS version again.
> So this is working as expected. ie- the same version should be seen
> and no other ISO should be available. Using the same iso may not require
> a reboot as well.

could i understand the follow according to your above comments?
(a) rhevh won't reboot after Re-install action finished
(b) There was only one version was shown during booting after reboot manually

> 
> For upgrade scenario I'd expect an additional ISO, but this is not
> what you describe. 
> 
> Can you please explain where and what is the problem?

The problem is Re-install action seems didn't execute after clicking "Reinstall" button. I didn't see any log info record in rhevh host about re-install action.
Something like upgrade scenario, it will use ovirt-node-upgrade script of rhevh and record upgrade log info in ovirt-node.log in /var/log/ of rhevh host

Can you explain the judgment standard for reinstall action success?

Comment 15 Douglas Schilling Landgraf 2014-12-29 13:30:44 UTC
(In reply to haiyang,dong from comment #14)
> (In reply to Doron Fediuck from comment #13)
> > Haiyang,
> > Re-install is installing the same OS version again.
> > So this is working as expected. ie- the same version should be seen
> > and no other ISO should be available. Using the same iso may not require
> > a reboot as well.
> 
> could i understand the follow according to your above comments?
> (a) rhevh won't reboot after Re-install action finished

Yes, it's like a new deploy action.

engine.log when clicking Re-install
====================================
<snip>
2014-12-29 06:59:50,116 INFO  [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-7-thread-15) [58e65425] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected :  ID: 7d834215-d306-4088-b1c3-1750904a1860 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
2014-12-29 06:59:52,780 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-55) Updated vds status from Preparing for Maintenance to Maintenance in database,  vds = 7d834215-d306-4088-b1c3-1750904a1860 : 192.168.122.217

* Node in maintenance

2014-12-29 06:59:59,622 INFO  [org.ovirt.engine.core.bll.InstallVdsCommand] (ajp-/127.0.0.1:8702-5) [22265408] Running command: InstallVdsCommand internal: false. Entities affected :  ID: 7d834215-d306-4088-b1c3-1750904a1860 Type: VDSAction group EDIT_HOST_CONFIGURATION with role type ADMIN

* Running InstallVdsCommand.
</snip>

However, please note that if user click Upgrade it executes UpgradeOvirtNodeCommand call which in the end will executing the upgrade tool on node and reboot.

Example from engine.log:
[org.ovirt.engine.core.bll.UpgradeOvirtNodeCommand] (ajp-/127.0.0.1:8702-7) [6e746da] Running command: UpgradeOvirtNodeCommand internal: false. Entities affected :  ID: 7d834215-d306-4088-b1c3-1750904a1860 Type: VDSAction group EDIT_HOST_CONFIGURATION with role type ADMIN


> (b) There was only one version was shown during booting after reboot manually

Correct.

> 
> > 
> > For upgrade scenario I'd expect an additional ISO, but this is not
> > what you describe. 
> > 
> > Can you please explain where and what is the problem?
> 
> The problem is Re-install action seems didn't execute after clicking
> "Reinstall" button. I didn't see any log info record in rhevh host about
> re-install action.
> Something like upgrade scenario, it will use ovirt-node-upgrade script of
> rhevh and record upgrade log info in ovirt-node.log in /var/log/ of rhevh
> host
> 
> Can you explain the judgment standard for reinstall action success?

I would use the same process of deploy. I would check /var/log/ovirt-engine/host-deploy logs. Is that clarify your report?

If you believe we could execute a different flow, I do believe it should go in a different bugzilla discussion with ovirt-engine guys.

Comment 16 Douglas Schilling Landgraf 2014-12-29 15:08:30 UTC
Hi hadong,

I am closing this bug as notabug now, if still have questions or if you think that's not correct state, fell free to re-open.

Thanks!

Comment 17 Ying Cui 2015-01-07 17:22:05 UTC
Douglas and Haiyang, 
  I am still a little confused on this bug, could you help to explain more below:

1. this "reinstall" button in rhevm portal, the behavior is not the same as "reinstall" on rhevh itself which remove hostvg, re-partition, reboot..., it is a little confused on QE, and consider it also be confused on customer.

2. in comment 11, "In the UI the reinstall used the current ISO", where is current iso setting?

3. in comment 15, "new deploy action", here deploy what? is it node maintenance? I am not sure why user need to do this new deploy?

4. in comment 15, "check /var/log/ovirt-engine/host-deploy logs.", that is not normal customer behavior to check log in server after new deploy. how the user know the new deploy is successful on portal?
  
   Thanks for help to explain more, we need to make this more clearly, then we design exact test cases to cover.

Thanks
Ying

Comment 18 Douglas Schilling Landgraf 2015-01-19 19:15:59 UTC
(In reply to Ying Cui from comment #17)
> Douglas and Haiyang, 
>   I am still a little confused on this bug, could you help to explain more
> below:
> 
> 1. this "reinstall" button in rhevm portal, the behavior is not the same as
> "reinstall" on rhevh itself which remove hostvg, re-partition, reboot..., it
> is a little confused on QE, and consider it also be confused on customer.
> 
> 2. in comment 11, "In the UI the reinstall used the current ISO", where is
> current iso setting?
> 
> 3. in comment 15, "new deploy action", here deploy what? is it node
> maintenance? I am not sure why user need to do this new deploy?
> 
> 4. in comment 15, "check /var/log/ovirt-engine/host-deploy logs.", that is
> not normal customer behavior to check log in server after new deploy. how
> the user know the new deploy is successful on portal?

If not sucessful, it will show in the event log. As we are using host-deploy to do the re-deploy the logs should be checked from there.

Alon, could you please give a brief info about host-deploy actions for the re-deploy on RHEV-H instead of full upgrade to QE guys?

Thanks!

Comment 19 Alon Bar-Lev 2015-01-19 19:19:33 UTC
in 3.5 there are two options in the menu: upgrade and reinstall


Note You need to log in before you can comment on or make changes to this bug.