Bug 1121182

Summary: [nova][live-migration] Live-migration fails with error "UnboundLocalError: local variable 'instance_dir' referenced before assignment"
Product: Red Hat OpenStack Reporter: Gabriel Szasz <gszasz>
Component: openstack-novaAssignee: Nikola Dipanov <ndipanov>
Status: CLOSED ERRATA QA Contact: Gabriel Szasz <gszasz>
Severity: high Docs Contact:
Priority: unspecified    
Version: 5.0 (RHEL 6)CC: dasmith, eglynn, jkleinpe, ndipanov, pbrady, sclewis, sgordon, vpopovic, yeylon
Target Milestone: rc   
Target Release: 5.0 (RHEL 7)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-nova-2014.1.1-4.el7ost Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-07-25 13:25:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1118488, 1152637, 1160405, 1160407, 1160408, 1180607, 1180608, 1183796, 1251236    
Attachments:
Description Flags
/var/log/nova/compute.log on the source node
none
/var/log/nova/compute.log on the target node none

Description Gabriel Szasz 2014-07-18 15:01:56 UTC
Description of problem:
Live-migration with shared nfs storage fails with error "UnboundLocalError: local variable 'instance_dir' referenced before assignment".

Version-Release number of selected component (if applicable):
openstack-nova-2014.1.1-1.el6ost.noarch
(RH6-RHOS5 repo, 2014-07-14.2 puddle)

How reproducible:
100%

Steps to Reproduce:
1. Deploy openstack with packstack in 3 node setup (controller + 2 compute nodes). Use another RHEL 6.5 system as an external storage

2. Create nova user and nova group on the external storage and set UID and GID to be the same as the one on the compute nodes. 

3. Export shared directory and restart NFS service on external storage

4. Mount the NFS shared directory on both compute nodes.

  mount $EXTERNAL_STORAGE:/export/instances /export/instances;

5. Set required options in nova.conf on both compute nodes:

   instances_path = /export/instances
   novncproxy_base_url = "http://$CONTROLLER_IP:6080/vnc_auto.html"
   vncserver_listen = '0.0.0.0'
   live_migration_flag = VIR_MIGRATE_UNDEFINE_SOURCE, VIR_MIGRATE_PEER2PEER, VIR_MIGRATE_LIVE'

6. Set required options in libvirtd.conf on both compute nodes

  listen_tls = 0
  listen_tcp = 1
  auth_tcp = "none"

7. Set required options in qemu.conf on both compute nodes

  user = "root"
  group = "root"
  vnc_listen = "0.0.0.0"

8. Set required options in nova.conf on controller:

   vncserver_listen '0.0.0.0'

9. Turn off firewall and SELinux on all nodes as far as this bug manifests itself regardless of firewall and SELinux settings.

10. Restart all relevant services on all nodes

11. Spawn an instance

12. Check actual compute node where the instance is running. Use the other one in the next step.

13. Run command:

  nova live-migration $INSTANCE $COMPUTE_NODE


Actual results:

Instance is not migrated to $COMPUTE_NODE. No error is being shown in CLI. 

/var/log/nova/compute.log shows an error: 

UnboundLocalError: local variable 'instance_dir' referenced before assignment


Expected results:

Instance is migrated, No error in /var/log/nova/compute


Additional info:

The bug was already reported in the upstream:
https://bugs.launchpad.net/nova/+bug/1281295

Comment 1 Gabriel Szasz 2014-07-18 15:06:04 UTC
Created attachment 919114 [details]
/var/log/nova/compute.log on the source node

Comment 2 Gabriel Szasz 2014-07-18 15:06:57 UTC
Created attachment 919115 [details]
/var/log/nova/compute.log on the target node

Comment 7 Dan Smith 2014-07-21 21:36:31 UTC
Fix patch proposed upstream:

https://review.openstack.org/#/c/108468

Comment 9 Scott Lewis 2014-07-25 13:25:53 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2014-0940.html

Comment 10 Pádraig Brady 2014-11-11 15:49:47 UTC
*** Bug 1160404 has been marked as a duplicate of this bug. ***