Description of problem: Failed multipath lose path during migration. Version-Release number of selected component (if applicable): RHEL OSP 8 How reproducible: Everytime for customer. Steps to Reproduce: 1. Attach a iscsi multipath cinder volume to an instance. 2. Simulate failure of two paths from 4 paths, live-migrate the instance. 3. On destination compute node failed paths disappeared for migrated instance and for the instance which is already present on destination compute node. 4. Re-enable the failed paths. Nothing changed in output. 5. After the re-scan only for the migrated instance paths re-appeared for the existing instance on destination node only two paths remained. Actual results: Failed multipaths are not showing after the migration. Expected results: Failed multipaths should show on destination compute node after the migration. Additional info: More info is coming in an internal comment.
Created attachment 1342055 [details] nova node 0 log file
Created attachment 1342057 [details] nova node 1 log file
While trying to verify the following bug I'm getting a lot of errors from nova. The log files of both nodes are attached here. The command running to do live migration : server migrate --block-migration --live compute-1.localdomain inst2 running the same scenario without volume attached passed
While disabling multipath from nova.conf The migration successfully passed
I've spent a bunch of time looking into this, and the only Nova errors I've seen so far have been in response to Cinder errors. I'm satisfied there's no specific Nova issue to address here. We could probably do better in response to Cinder errors, but that would require a redesign which wouldn't be applicable to OSP8. Specifically, the most common failure to migrate I see manifests like this in the libvirt logs: 2017-10-31 13:56:24.555+0000: 1726: error : virNetClientProgramDispatchError:177 : internal error: qemu unexpectedly closed the monitor: 2017-10-31T13:56:24.146532Z qemu-kvm: -chardev pty,id=charserial1: char device redirected to /dev/pts/2 (labe l charserial1) 2017-10-31T13:56:24.342623Z qemu-kvm: load of migration failed: Input/output error 2017-10-31 13:56:24.555+0000: 1726: debug : qemuDomainObjExitRemote:4151 : Exited remote (vm=0x7feea4017790 name=instance-00000011) This appears to relate to a real failure of the multipath block device: # multipath -ll 3514f0c5a51600fd3 dm-0 XtremIO ,XtremApp size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='queue-length 0' prio=0 status=enabled |- 10:0:0:1 sda 8:0 failed faulty running |- 11:0:0:1 sdb 8:16 failed faulty running |- 12:0:0:1 sdc 8:32 failed faulty running `- 13:0:0:1 sdd 8:48 failed faulty running 3514f0c5a51600fd5 dm-2 XtremIO ,XtremApp size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='queue-length 0' prio=1 status=active |- 10:0:0:2 sdf 8:80 active ready running |- 11:0:0:2 sde 8:64 active ready running |- 12:0:0:2 sdg 8:96 active ready running `- 13:0:0:2 sdh 8:112 active ready running [root@compute-0 ~]# cat /dev/dm-0 > /dev/null cat: /dev/dm-0: Input/output error My assumption here is that qemu is failing with the same I/O error I get from cat. I've also seen 500 responses from nova-api, but in every case these have been in cinder client, with a corresponding error in cinder's volume.log. Unfortunately I don't have any useful insight into the cause of the multipath or cinder failures, so I can't say whether or not they're expected. However, Nova failures are expected if the volume is inaccessible by the host.
Verified Package version : openstack-cinder-7.0.3-10.el7ost.noarch server migration successfully executed with multipath configuration. One of two multipath interfaces traffic was block with iptables on all compute servers while instance migration.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:3283