Bug 1250556 - ovirt-engine-rename tool fails when local ISO domain is available.
ovirt-engine-rename tool fails when local ISO domain is available.
Status: CLOSED ERRATA
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine (Show other bugs)
3.5.3
All Linux
high Severity medium
: ovirt-3.6.3
: 3.6.3
Assigned To: Yedidyah Bar David
Gonza
:
Depends On: 1157758 1289983
Blocks: 1313217
  Show dependency treegraph
 
Reported: 2015-08-05 08:54 EDT by Ulhas Surse
Modified: 2016-03-09 16:10 EST (History)
16 users (show)

See Also:
Fixed In Version:
Doc Type: Known Issue
Doc Text:
Previously, changing the host name of the Manager machine when the machine also provided a local ISO storage domain was prevented by the rename tool, to prevent the ISO storage domain losing connectivity during the renaming process. Also, changing the host name of the Manager machine when the machine also provided a local data domain could cause running virtual machines to become stuck due to the hosts losing connectivity with their virtual disks. Now, the rename tool checks both of these, and guides the user to eject/shut down/put to maintenance any required virtual machines or storage domains, and only then allows the rename process to continue. To change the host name of the Manager machine, see the product documentation: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Virtualization/3.6/html-single/Administration_Guide/index.html#sect-The_oVirt_Engine_Rename_Tool
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-03-09 16:10:45 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Integration
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 50394 master MERGED packaging: rename: change also ENGINE_FQDN 2015-12-21 12:20 EST
oVirt gerrit 50395 master MERGED packaging: rename: Move new fqdn prompt to customization 2015-12-22 02:45 EST
oVirt gerrit 50396 master MERGED packaging: rename: Handle storage domains more nicely 2016-02-02 03:41 EST
oVirt gerrit 50946 ovirt-engine-3.6 MERGED packaging: rename: change also ENGINE_FQDN 2015-12-23 12:57 EST
oVirt gerrit 50947 ovirt-engine-3.6 MERGED packaging: rename: Move new fqdn prompt to customization 2015-12-23 12:58 EST
oVirt gerrit 50948 ovirt-engine-3.6.2 MERGED packaging: rename: change also ENGINE_FQDN 2015-12-23 13:00 EST
oVirt gerrit 50949 ovirt-engine-3.6.2 MERGED packaging: rename: Move new fqdn prompt to customization 2015-12-23 13:01 EST
oVirt gerrit 52898 ovirt-engine-3.6 MERGED packaging: rename: Handle storage domains more nicely 2016-02-02 14:06 EST
oVirt gerrit 52899 ovirt-engine-3.6.3 MERGED packaging: rename: Handle storage domains more nicely 2016-02-02 14:07 EST

  None (edit)
Description Ulhas Surse 2015-08-05 08:54:29 EDT
Description of problem:
While running the script to rename the RHEVM with ovirt-engine-rename tool, following error observed when ISO domain exist on RHEVM system.

======
2015-08-04 10:05:59 DEBUG otopi.ovirt_**FILTERED**_setup.**FILTERED**_common.database database.execute:214 Result: [{'connection': '<rhevm_hostname>:/_iso_domain', 'storage_name': 'ISOs'}]
2015-08-04 10:05:59 WARNING otopi.plugins.ovirt_**FILTERED**_rename.ovirt-**FILTERED**.database database._validation:138 Engine host hosting Storage Domains
2015-08-04 10:05:59 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:215 DIALOG:SEND                 The following Storage Domains use the **FILTERED** host
2015-08-04 10:05:59 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:215 DIALOG:SEND                 as an NFS server:
2015-08-04 10:05:59 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:215 DIALOG:SEND                
2015-08-04 10:05:59 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:215 DIALOG:SEND                 ISOs
2015-08-04 10:05:59 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:215 DIALOG:SEND                
2015-08-04 10:05:59 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:215 DIALOG:SEND                 Cannot rename the **FILTERED** host. Please backup relevant
2015-08-04 10:05:59 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:215 DIALOG:SEND                 data if needed, remove all of these domains, and then
2015-08-04 10:05:59 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:215 DIALOG:SEND                 run this utility again.
2015-08-04 10:05:59 DEBUG otopi.context context._executeMethod:152 method exception
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/otopi/context.py", line 142, in _executeMethod
    method['method']()
  File "/usr/share/ovirt-**FILTERED**/setup/bin/../plugins/ovirt-**FILTERED**-rename/ovirt-**FILTERED**/database.py", line 153, in _validation
    raise RuntimeError(_('Cannot rename host hosting Storage Domains'))
RuntimeError: Cannot rename host hosting Storage Domains
======

Version-Release number of selected component (if applicable):
RHEVM 3.5.3

How reproducible:
Always

Steps to Reproduce:
1. Create local on host ISO domain
2. run ovirt-engine-rename utility
3. script fails

Actual results:
script fails with error:  Cannot rename host hosting Storage Domains

Expected results:
Script should run and change the domain hostname as well.


Additional info:
I am not sure about the component to select here.
Comment 2 Yedidyah Bar David 2015-10-27 08:57:59 EDT
No idea what needs to change, nor what's the best way to do that (directly in db? api/sdk? etc.). Also no idea what effect, if at all, this might have on running VMs etc. Also not sure who to ask :-) Sandro?
Comment 3 Sandro Bonazzola 2015-10-29 05:45:38 EDT
"Cannot rename host hosting Storage Domains" has been there since the rename tool has been created (commit 50eada928cda82144c3801da5ba7494f6287d777)

the referenced wiki http://www.ovirt.org/Changing_Engine_Hostname doesn't cover specifications about storage domains handling so I'm not sure about why the check has been added.

Issue here is that we rename the host and it's hosting shared storage, all the VMs will probably be paused loosing storage connection.

Adding allon to the loop.
Comment 4 Allon Mureinik 2015-10-29 07:43:24 EDT
(In reply to Sandro Bonazzola from comment #3)
> "Cannot rename host hosting Storage Domains" has been there since the rename
> tool has been created (commit 50eada928cda82144c3801da5ba7494f6287d777)
> 
> the referenced wiki http://www.ovirt.org/Changing_Engine_Hostname doesn't
> cover specifications about storage domains handling so I'm not sure about
> why the check has been added.
> 
> Issue here is that we rename the host and it's hosting shared storage, all
> the VMs will probably be paused loosing storage connection.
> 
> Adding allon to the loop.

If I understand correctly, the issue here is that you're renaming the engine's host name and aren't able to access storage domains located ON THAT HOST afterwards, right?

If this is indeed the case, the mount path to any NFS domain should also be update accordingly:


    UPDATE storage_server_connections
    SET    connection = REPLACE(connection, 'old-host-name', 'new-host-name'
    WHERE  vfs_type = 'nfs' AND connection LIKE CONCAT('old-host-name',':%')
Comment 5 Allon Mureinik 2015-10-29 07:44:33 EDT
Missing a bracket there, of course (copy-paste error, apologies):
(In reply to Allon Mureinik from comment #4)
>     UPDATE storage_server_connections
>     SET    connection = REPLACE(connection, 'old-host-name', 'new-host-name'
>     WHERE  vfs_type = 'nfs' AND connection LIKE CONCAT('old-host-name',':%')

     UPDATE storage_server_connections
     SET    connection = REPLACE(connection, 'old-host-name', 'new-host-name')
     WHERE  vfs_type = 'nfs' AND connection LIKE CONCAT('old-host-name', ':%')
Comment 6 Yedidyah Bar David 2015-11-01 02:46:16 EST
Thanks.

What happens to running VMs?

This tool is specifically for _renaming_, not renumbering. Can we safely assume
that the hosts will simply keep the domains mounted (with same IP address etc)
and will simply look up the new name when needed (e.g. on migration)? No restart
of anything needed?

I still think it will be wise to notify/query the user about this.
Comment 7 Allon Mureinik 2015-11-01 03:37:15 EST
(In reply to Yedidyah Bar David from comment #6)
> Can we safely
> assume
> that the hosts will simply keep the domains mounted (with same IP address
> etc)
> and will simply look up the new name when needed (e.g. on migration)? No
> restart
> of anything needed?
The host (and thus - VM) will attempt to keep the same mount paths active until the engine explicitly tells them to do something else (e.g., by putting the domain to maintenance).
Note that if the mount is done via the host NAME you might get a stale mount point on the host side.

Having said that, a VM shouldn't really read from an ISO domain after it's done booting.
Comment 8 Yaniv Lavi (Dary) 2015-11-02 09:59:17 EST
Can you please make sure to get docs on this for rename process?
We should allow forcing maybe with flag in a case were the user doesn't have any VM depending on the iso domain.
Comment 9 Sandro Bonazzola 2015-11-11 10:22:48 EST
Didi, can you update the oVirt wiki page about the rename tool[1] once fixed?

[1] http://www.ovirt.org/Changing_Engine_Hostname

Please also review Doc-Text.
Comment 10 Yedidyah Bar David 2015-11-16 06:03:34 EST
(In reply to Allon Mureinik from comment #7)
> (In reply to Yedidyah Bar David from comment #6)
> > Can we safely
> > assume
> > that the hosts will simply keep the domains mounted (with same IP address
> > etc)
> > and will simply look up the new name when needed (e.g. on migration)? No
> > restart
> > of anything needed?
> The host (and thus - VM) will attempt to keep the same mount paths active
> until the engine explicitly tells them to do something else (e.g., by
> putting the domain to maintenance).
> Note that if the mount is done via the host NAME you might get a stale mount
> point on the host side.

To be on the safe side, I think we should check some/all of these:
1. Is the storage domain active?
2. Is there a VM with it attached?

And if 'yes', query user, suggesting to stop VMs and/or put domain to maint.

Allon - how do I check these?

> 
> Having said that, a VM shouldn't really read from an ISO domain after it's
> done booting.

Users might use the iso domain also for other things, besides OSes. In particular we also supply (and expect people to use) the windows guest tools as an iso image. I agree that this one too is expected to be used mostly shortly after installation (or upgrades) but still.

Thanks.
Comment 11 Allon Mureinik 2015-11-16 11:04:02 EST
Amit - cam you answer this please? Thanks!
Comment 12 Amit Aviram 2015-11-16 12:15:28 EST
These are simple SQL queries for your question, tell me if you need anything further than that.

> 1. Is the storage domain active?
select CASE status WHEN 3 THEN 't' ELSE 'f' END from storage_domains where id = '%storageDomainID%'

> 2. Is there a VM with it attached?
SELECT exists(
SELECT * FROM all_disks_for_vms
WHERE storage_id = '%storageDomainID%'
)
Comment 13 Amit Aviram 2015-11-17 03:00:48 EST
(In reply to Amit Aviram from comment #12)
> These are simple SQL queries for your question, tell me if you need anything
> further than that.
> 
> > 1. Is the storage domain active?
> select CASE status WHEN 3 THEN 't' ELSE 'f' END from storage_domains where
> id = '%storageDomainID%'
> 
> > 2. Is there a VM with it attached?
> SELECT exists(
> SELECT * FROM all_disks_for_vms
> WHERE storage_id = '%storageDomainID%'
> )

Just to be clear, all that is marked with "%something%" is a placeholder for the actual UUID you are looking for, and it is not the actual syntax to use 
(e.g. use:
> select CASE status WHEN 3 THEN true ELSE false END from storage_domains where
> id = '477afe45-c8f5-4985-81ba-b9e2603bb6d9'
)

Also- for the 2nd query, you might want to get the VM names&id instead of just t/f. in that case you can use:

SELECT distinct vm_names, vm_id FROM all_disks_for_vms
WHERE storage_id = '%storageDomainID%'

(Allon, thanks for highlighting this)
Comment 14 Yaniv Kaul 2015-12-03 10:05:13 EST
Any updates to this bug (it's marked as high priority) ?
Comment 15 Yedidyah Bar David 2015-12-07 03:18:39 EST
(In reply to Yaniv Kaul from comment #14)
> Any updates to this bug (it's marked as high priority) ?

Working on it, hopefully will have something this week.
Comment 16 Yedidyah Bar David 2015-12-09 09:54:17 EST
Amit, the above covers only disks, not iso images. How can I check if a vm has an iso image from a specific iso domain attached?
Comment 17 Yedidyah Bar David 2015-12-09 10:22:08 EST
did 'pg_dump engine' before and after 'Change CD', and the only relevant change seems to me to be in vm_dynamic.current_cd .
Comment 18 Yedidyah Bar David 2015-12-09 14:52:47 EST
Amit told me in private to check vm_static.iso_path. Amit - just a guess, judging the names - isn't it that 'vm_static.iso_path' is the image the vm is configured to boot from, whereas 'vm_dynamic.current_cd' is the currently attached cd?
Comment 19 Amit Aviram 2015-12-10 04:05:37 EST
Meanings of theses fields are not reflected so much from its names :)

"vm_static.iso_path" describes the image that will be attached to the VM as a CD-ROM *when it will be activated*

"vm_dynamic.current_cd" describes the images that was connected to the VM as a CD *When it was activated*

For example- 
- I am attaching some image "disk.iso" which resides in my ISO domain to a turned off VM. now its "vm_static.iso_path" will have "disk.iso" as a value.
- I am activating the VM. now "vm_dynamic.current_cd" will have "disk.iso" as a value.
- I am editing the VM *while it is up*, and removing the CD-ROM. this will make change only when I'll restart the VM, vm_static.iso_path will contain nothing as a value, but "vm_dynamic.current_cd" will still have "disk.iso" as a value.

Hope that explains the meaning better :)
Comment 20 Yedidyah Bar David 2016-01-19 10:53:31 EST
Not sure how to continue working on this. Patch is currently pending review.

There are two questions here:

1. What to check

For vm status, patch currently allows only '0'. So paused will still emit a message.

2. whether to allow forcing a rename even if the checks fail

Personally I do not like software that thinks it's smarter than me ...

Is it absolutely mandatory?

E.g. if a user forces a rename, waits until hosts/VMs are stuck, then forcefully reboots them, should there be any long-term problems, apart from normal problems that might happen when plugging out the power cord?
Comment 21 Yedidyah Bar David 2016-01-20 08:06:35 EST
Talked with Amit about this. Summary:

1. Basically the checks are ok.

2. The tool will not allow forcing a rename if any checks fail. We'll change that if (when?) people complain about this with valid cases.
Comment 22 Tal Nisan 2016-01-20 08:08:02 EST
IIUC you should probably also make sure that the storage domains are moved to maintenance otherwise the old mount points will be used which now are pointing to a non existing host name
Comment 23 Yedidyah Bar David 2016-01-20 08:19:22 EST
(In reply to Tal Nisan from comment #22)
> IIUC you should probably also make sure that the storage domains are moved
> to maintenance otherwise the old mount points will be used which now are
> pointing to a non existing host name

AFAIU I already do that, you are welcome to review the patch. Thanks!
Comment 24 Amit Aviram 2016-01-20 08:37:45 EST
(In reply to Tal Nisan from comment #22)
> IIUC you should probably also make sure that the storage domains are moved
> to maintenance otherwise the old mount points will be used which now are
> pointing to a non existing host name

That's also being taken care of. thanks!
Comment 25 Yedidyah Bar David 2016-02-03 03:06:05 EST
Adapted the doc text to the current behavior. Andrew/QE - please search for "Verified+1" comments in [1] for an example flow.

[1] https://gerrit.ovirt.org/50396
Comment 26 Yedidyah Bar David 2016-02-03 03:08:20 EST
Andrew - you probably should also update the actual documentation. Setting needinfo on you as a reminder - you can open a doc bug if needed.
Comment 27 Gonza 2016-02-19 05:28:40 EST
Verified with:
rhevm-3.6.3.1-0.1.el6.noarch

[ INFO  ] Engine machine hosting Storage Domains
          The following Storage Domains use the engine machine as an NFS server, and are active:
         
          ISO_DOMAIN
         
          Needed action: They should be moved to Maintenance.
         
          Some storage domains hosted on the engine machine are in use. Please complete the above detailed actions.
          To abort, simply kill this utility with ^C.
          Press Enter to continue: 
[ INFO  ] Engine machine hosting Storage Domains
          The following Storage Domains use the engine machine as an NFS server:
         
          ISO_DOMAIN
         
          They will be modified to use the new name.
Comment 28 Lucy Bopf 2016-03-03 02:16:52 EST
Thanks, Didi, for bringing the fix to our attention. I've raised bug 1313217 to track the addition of a note to our existing ovirt-engine-rename content.

Should the 'Doc Type' on this bug also be moved from 'Known Issue' to 'Bug Fix' now?
Comment 29 Yedidyah Bar David 2016-03-03 03:48:53 EST
Done.
Comment 31 errata-xmlrpc 2016-03-09 16:10:45 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-0376.html

Note You need to log in before you can comment on or make changes to this bug.