Bug 1312209

Summary: Nova is unable to find iSCSI volume after rebooting the compute node
Product: Red Hat OpenStack Reporter: Chen <cchen>
Component: openstack-novaAssignee: Lee Yarwood <lyarwood>
Status: CLOSED INSUFFICIENT_DATA QA Contact: nlevinki <nlevinki>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.0 (Juno)CC: berrange, cchen, dasmith, eglynn, kchamart, lyarwood, sbauza, sferdjao, sgordon, srevivo, vromanso
Target Milestone: ---Keywords: ZStream
Target Release: 6.0 (Juno)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-06-27 13:39:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Chen 2016-02-26 06:23:22 UTC 
Description of problem:

Nova is unable to find iSCSI volume after rebooting the compute node

Version-Release number of selected component (if applicable):

OSP 6.0

How reproducible:


Steps to Reproduce:
1. Boot an instance from volume (iSCSI backend)
2. Reboot the compute node
3. Resume the instance

Actual results:

The instance failed to resume and the following error could be found:

2016-02-13 16:03:31.230 4323 WARNING nova.virt.libvirt.volume [req-d051fda8-0098-4b3a-814e-86f503bb4fb0 None] ISCSI volume not yet found at: vda. Will rescan & retry.  Try number: 0
2016-02-13 16:03:32.292 4323 WARNING nova.virt.libvirt.volume [req-d051fda8-0098-4b3a-814e-86f503bb4fb0 None] ISCSI volume not yet found at: vda. Will rescan & retry.  Try number: 1
2016-02-13 16:03:36.363 4323 WARNING nova.virt.libvirt.volume [req-d051fda8-0098-4b3a-814e-86f503bb4fb0 None] ISCSI volume not yet found at: vda. Will rescan & retry.  Try number: 2
2016-02-13 16:03:45.433 4323 WARNING nova.virt.libvirt.volume [req-d051fda8-0098-4b3a-814e-86f503bb4fb0 None] ISCSI volume not yet found at: vda. Will rescan & retry.  Try number: 3
2016-02-13 16:04:01.503 4323 WARNING nova.virt.libvirt.volume [req-d051fda8-0098-4b3a-814e-86f503bb4fb0 None] ISCSI volume not yet found at: vda. Will rescan & retry.  Try number: 4
2016-02-13 16:04:26.701 4323 ERROR oslo.messaging.rpc.dispatcher [req-d051fda8-0098-4b3a-814e-86f503bb4fb0 ] Exception during message handling: iSCSI device not found at [u'/dev/disk/by-path/ip-192.168.10.170:3260-iscsi-iqn.2010-10.org.openstack:volume-cab83bd2-bb3d-4b63-85ba-2fff11d5d448-lun-0']

Expected results:

The instance should be successfully resume.

Additional info:

sosreports for both controller and compute (after the accident) are in the 

collab-shell.usersys.redhat.com:/cases/01583215
Comment 2 Chen 2016-03-11 08:48:01 UTC 
Hi Lee,

Do you think this is something like "slow scanning disk" issue within compute node ? Or is there anywhere suspicious ?

The customer got a plan to reboot the environment again and they are worrying that they could encounter the issue again. I have asked them to collect the sosreports when the problem is occurring. Is there anything you need ?

Best regards,
Chen
Comment 3 Lee Yarwood 2016-04-26 12:57:09 UTC 
(In reply to Chen from comment #2)
> Hi Lee,
> 
> Do you think this is something like "slow scanning disk" issue within
> compute node ? Or is there anywhere suspicious ?
> 
> The customer got a plan to reboot the environment again and they are
> worrying that they could encounter the issue again. I have asked them to
> collect the sosreports when the problem is occurring. Is there anything you
> need ?

Hello Chen, 

Apologies for the delay, it appears you forgot to add a needinfo.

(In reply to Chen from comment #0)
> Steps to Reproduce:
> 1. Boot an instance from volume (iSCSI backend)
> 2. Reboot the compute node
> 3. Resume the instance

I'm unable to reproduce this using the latest 2014.2.3 builds of Nova.

Would the customer be willing to update to 2014.2.3, increase num_iscsi_scan_tries, enable debug logging and reproduce?