Bug 1394964 - Live migration with config-drive fails with InvalidSharedStorage error
Summary: Live migration with config-drive fails with InvalidSharedStorage error
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova
Version: 7.0 (Kilo)
Hardware: Unspecified
OS: Linux
unspecified
high
Target Milestone: async
: 7.0 (Kilo)
Assignee: melanie witt
QA Contact: awaugama
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-11-14 21:53 UTC by nalmond
Modified: 2022-07-09 08:40 UTC (History)
13 users (show)

Fixed In Version: openstack-nova-2015.1.4-20.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-02-15 23:00:01 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker OSP-16807 0 None None None 2022-07-09 08:40:17 UTC
Red Hat Product Errata RHSA-2017:0282 0 normal SHIPPED_LIVE Moderate: openstack-cinder, openstack-glance, and openstack-nova security update 2017-02-16 03:52:44 UTC

Description nalmond 2016-11-14 21:53:02 UTC
Description of problem:
Instances booted with a config-drive are not able to be live migrated

Version-Release number of selected component (if applicable):

How reproducible:
Everytime

Steps to Reproduce:
1. Boot instance with --config-drive flag
2. Attempt a live migration

Actual results:
Migration fails with 'InvalidSharedStorage: <host> is not on shared storage: Live migration can not be used without shared storage.'

Expected results:
Migration is successful

Additional info:
Instances booted without a config drive migrate without issue.
When the migration is attempted with the '--block-migrate' flag, it fails with 'Cannot block migrate instance <uuid> with mapped volumes'

Comment 2 melanie witt 2016-11-18 23:19:09 UTC
In the environment, has Nova been configured to use shared storage? That is, are state_path and instance_path in nova.conf set to directories that are shared among the Nova compute hosts?

Comment 3 melanie witt 2016-11-19 02:53:17 UTC
After researching this more, I think the problem is that live migration of volume backed instances with local disk (config drive) is being attempted, which is not supported.


I think there are two options for getting this particular live migration to work.

  1. Store config drives on shared storage. This will avoid the combination of volume backed + local disk for live migration.

  or

  2. Upgrade to the latest OSP 7 (openstack-nova-2015.1.4-21.el7ost) and use block live migration (--block-migrate option) with config_drive_format=vfat. They may also need to turn tunnelling off but I would first try without changing it (default is tunnelling on).

The latest version of OSP 7 contains a feature "Allow block live migration of an instance with attached volumes" which isn't present in the older version (openstack-nova-compute-2015.1.4-17.el7ost) they currently have. That's why they get the error "Cannot block migrate instance <uuid> with mapped volumes" when they try --block-migrate.

Comment 4 Stephen Gordon 2016-11-19 19:33:37 UTC
(In reply to melanie witt from comment #3)
> After researching this more, I think the problem is that live migration of
> volume backed instances with local disk (config drive) is being attempted,
> which is not supported.

Specifically not supported in 7 right? I seem to recall it will work in a later version (9?) due to https://bugs.launchpad.net/nova/+bug/1246201

Comment 5 melanie witt 2016-11-21 17:25:54 UTC
(In reply to Stephen Gordon from comment #4)
> (In reply to melanie witt from comment #3)
> > After researching this more, I think the problem is that live migration of
> > volume backed instances with local disk (config drive) is being attempted,
> > which is not supported.
> 
> Specifically not supported in 7 right? I seem to recall it will work in a
> later version (9?) due to https://bugs.launchpad.net/nova/+bug/1246201

There's a distinction between "live migration" and "block live migration." Live migration requires that all storage be shared between the source and the destination [1]. Block live migration (--block-migrate option) does not require shared storage and will handle copying of non-read-only local disks from the source to the destination [2]. This includes vfat (read-write) config drives that are local disks. There is a patch that enables block live migration of instances with local iso9660 (read-only) config drives that was backported to Mitaka [3] and not available in OSP 7. In OSP 7, only vfat config drives can be handled.

So even in OSP 9, I don't believe "live migration" of an instance with a local disk will work, whereas "block live migration" will work.

"Block live migration" (--block-migrate) will work in OSP 7 too but the latest version is needed because it contains the feature that will handle skipping the copy of the attached volumes. Volumes need to be skipped during the copy otherwise it would copy volumes onto themselves, risking corruption.

[1] http://docs.openstack.org/admin-guide/compute-configuring-migrations.html#shared-storage
[2] http://docs.openstack.org/admin-guide/compute-configuring-migrations.html#block-migration
[3] https://review.openstack.org/234659

Comment 7 melanie witt 2016-11-30 19:13:45 UTC
(In reply to nalmond from comment #6)
> The /var/lib/nova directories on the compute nodes are not on shared
> storage, they are on local storage. However, when testing in OSP 9, they
> were able to perform a live migration with the config drive (vfat) on local
> storage without --block-migration, output below.

Hm, that is unexpected. The --block-migrate option is supposed to be needed in order to copy any local disks (config drive) to the destination host.

> Regarding the nova version
> for OSP 7, they are already running a version newer than the latest one on
> the customer portal (2015.1.4-15.el7ost). Do you know when
> 2015.1.4-21.el7ost will be released, or if we can get them a hotfix build?

Okay, it turns out 2015.1.4-21.el7ost hasn't been released yet, but the needed feature was added in 2015.1.4-20.el7ost, which has already been tagged as a hotfix build to solve a similar bug [1]. As mentioned in the other bug, python-oslo-concurrency-1.8.2-2.el7ost might also be needed as a dependency.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1380289#c15

Comment 8 melanie witt 2016-11-30 20:06:34 UTC
(In reply to melanie witt from comment #7)
> Hm, that is unexpected. The --block-migrate option is supposed to be needed
> in order to copy any local disks (config drive) to the destination host.

I found in OSP 9 there's code that will set the internal block_migration flag automatically if shared storage is not detected, so that's how it works without --block-migrate:

 if 'block_migration' not in dest_check_data:
     dest_check_data.block_migration = (
         not dest_check_data.is_on_shared_storage())

Comment 9 nalmond 2016-12-12 19:28:46 UTC
After updating to openstack-nova-2015.1.4-20.el7ost and removing VIR_MIGRATE_TUNNELLED from the block_migration_flag in nova.conf, the instances were able to be live migrated with block-migration.

Comment 16 errata-xmlrpc 2017-02-15 23:00:01 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2017-0282.html


Note You need to log in before you can comment on or make changes to this bug.