In the environment, has Nova been configured to use shared storage? That is, are state_path and instance_path in nova.conf set to directories that are shared among the Nova compute hosts?

After researching this more, I think the problem is that live migration of volume backed instances with local disk (config drive) is being attempted, which is not supported.


I think there are two options for getting this particular live migration to work.

  1. Store config drives on shared storage. This will avoid the combination of volume backed + local disk for live migration.

  or

  2. Upgrade to the latest OSP 7 (openstack-nova-2015.1.4-21.el7ost) and use block live migration (--block-migrate option) with config_drive_format=vfat. They may also need to turn tunnelling off but I would first try without changing it (default is tunnelling on).

The latest version of OSP 7 contains a feature "Allow block live migration of an instance with attached volumes" which isn't present in the older version (openstack-nova-compute-2015.1.4-17.el7ost) they currently have. That's why they get the error "Cannot block migrate instance <uuid> with mapped volumes" when they try --block-migrate.

(In reply to melanie witt from comment #3)
> After researching this more, I think the problem is that live migration of
> volume backed instances with local disk (config drive) is being attempted,
> which is not supported.

Specifically not supported in 7 right? I seem to recall it will work in a later version (9?) due to https://bugs.launchpad.net/nova/+bug/1246201

(In reply to Stephen Gordon from comment #4)
> (In reply to melanie witt from comment #3)
> > After researching this more, I think the problem is that live migration of
> > volume backed instances with local disk (config drive) is being attempted,
> > which is not supported.
> 
> Specifically not supported in 7 right? I seem to recall it will work in a
> later version (9?) due to https://bugs.launchpad.net/nova/+bug/1246201

There's a distinction between "live migration" and "block live migration." Live migration requires that all storage be shared between the source and the destination [1]. Block live migration (--block-migrate option) does not require shared storage and will handle copying of non-read-only local disks from the source to the destination [2]. This includes vfat (read-write) config drives that are local disks. There is a patch that enables block live migration of instances with local iso9660 (read-only) config drives that was backported to Mitaka [3] and not available in OSP 7. In OSP 7, only vfat config drives can be handled.

So even in OSP 9, I don't believe "live migration" of an instance with a local disk will work, whereas "block live migration" will work.

"Block live migration" (--block-migrate) will work in OSP 7 too but the latest version is needed because it contains the feature that will handle skipping the copy of the attached volumes. Volumes need to be skipped during the copy otherwise it would copy volumes onto themselves, risking corruption.

[1] http://docs.openstack.org/admin-guide/compute-configuring-migrations.html#shared-storage
[2] http://docs.openstack.org/admin-guide/compute-configuring-migrations.html#block-migration
[3] https://review.openstack.org/234659

(In reply to nalmond from comment #6)
> The /var/lib/nova directories on the compute nodes are not on shared
> storage, they are on local storage. However, when testing in OSP 9, they
> were able to perform a live migration with the config drive (vfat) on local
> storage without --block-migration, output below.

Hm, that is unexpected. The --block-migrate option is supposed to be needed in order to copy any local disks (config drive) to the destination host.

> Regarding the nova version
> for OSP 7, they are already running a version newer than the latest one on
> the customer portal (2015.1.4-15.el7ost). Do you know when
> 2015.1.4-21.el7ost will be released, or if we can get them a hotfix build?

Okay, it turns out 2015.1.4-21.el7ost hasn't been released yet, but the needed feature was added in 2015.1.4-20.el7ost, which has already been tagged as a hotfix build to solve a similar bug [1]. As mentioned in the other bug, python-oslo-concurrency-1.8.2-2.el7ost might also be needed as a dependency.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1380289#c15

(In reply to melanie witt from comment #7)
> Hm, that is unexpected. The --block-migrate option is supposed to be needed
> in order to copy any local disks (config drive) to the destination host.

I found in OSP 9 there's code that will set the internal block_migration flag automatically if shared storage is not detected, so that's how it works without --block-migrate:

 if 'block_migration' not in dest_check_data:
     dest_check_data.block_migration = (
         not dest_check_data.is_on_shared_storage())

After updating to openstack-nova-2015.1.4-20.el7ost and removing VIR_MIGRATE_TUNNELLED from the block_migration_flag in nova.conf, the instances were able to be live migrated with block-migration.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2017-0282.html