Bug 1027680

Summary: Destroy a NPIV pool which lack of adapter's parent lead to libvirtd crash
Product: Red Hat Enterprise Linux 7 Reporter: Luwen Su <lsu>
Component: libvirtAssignee: John Ferlan <jferlan>
Status: CLOSED CURRENTRELEASE QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.0CC: acathrow, dyuan, gsun, juzhang, shyu
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-1.1.1-13.el7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-06-13 09:30:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Luwen Su 2013-11-07 09:35:10 UTC
Description of problem:
Destroy a pool which lack of elemet parent lead to libvirtd crash

Version-Release number of selected component (if applicable):
libvirt-1.1.1-11.el7.x86_64
kernel-3.10.0-41.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Prepare the pool's xml
    <pool type='scsi'>
      <name>test</name>
      <source>
        <adapter type='fc_host'   wwnn='20000024ff370144'wwpn='20000024ff370144'/>
      </source>
      <target>
        <path>/dev/disk/by-path/</path>
        <permissions>
          <mode>0755</mode>
          <owner>-1</owner>
          <group>-1</group>
        </permissions>
      </target>
    </pool>


the wwnn and wwpn pointed to a vport in my host's 
 #virsh nodedev-dumpxml scsi_host1
<device>
  <name>scsi_host1</name>
  <path>/sys/devices/pci0000:00/0000:00:09.0/0000:18:00.0/host1</path>
  <parent>pci_0000_18_00_0</parent>
  <capability type='scsi_host'>
    <host>1</host>
    <capability type='fc_host'>
      <wwnn>20000024ff370144</wwnn>
      <wwpn>20000024ff370144</wwpn>
      <fabric_wwn>2001000dec9877c1</fabric_wwn>
    </capability>
    <capability type='vport_ops'>
      <max_vports>254</max_vports>
      <vports>0</vports>
    </capability>
  </capability>
</device>

2.do
#virsh pool-define pool.xml
#virsh pool-start test
#virsh pool-destroy test
error: Failed to destroy pool test
error: End of file while reading data: Input/output error
error: One or more references were leaked after disconnect from the hypervisor
error: Failed to reconnect to the hypervisor


3.Gdb's backtrace

(gdb)thread apply all backtrace
Thread 11 (Thread 0x7f75cdc00700 (LWP 16443)):
#0  getHostNumber (adapter_name=adapter_name@entry=0x0, result=result@entry=0x7f75cdbffacc) at storage/storage_backend_scsi.c:579
#1  0x00007f75c87bd43f in deleteVport (adapter=...) at storage/storage_backend_scsi.c:671
#2  virStorageBackendSCSIStopPool (conn=<optimized out>, pool=<optimized out>) at storage/storage_backend_scsi.c:755
#3  0x00007f75c87b11e5 in storagePoolDestroy (obj=0x7f75a8000c50) at storage/storage_driver.c:874
#4  0x00007f75dc53b8a6 in virStoragePoolDestroy (pool=pool@entry=0x7f75a8000c50) at libvirt.c:13859
#5  0x00007f75dcf2cfa8 in remoteDispatchStoragePoolDestroy (server=<optimized out>, msg=<optimized out>, args=<optimized out>, rerr=0x7f75cdbffc90, 
    client=0x7f75deae5fd0) at remote_dispatch.h:12412
#6  remoteDispatchStoragePoolDestroyHelper (server=<optimized out>, client=0x7f75deae5fd0, msg=<optimized out>, rerr=0x7f75cdbffc90, args=<optimized out>, 
    ret=<optimized out>) at remote_dispatch.h:12390
#7  0x00007f75dc587a4a in virNetServerProgramDispatchCall (msg=0x7f75deae4830, client=0x7f75deae5fd0, server=0x7f75dead6c70, prog=0x7f75deae0570)
    at rpc/virnetserverprogram.c:435
#8  virNetServerProgramDispatch (prog=0x7f75deae0570, server=server@entry=0x7f75dead6c70, client=0x7f75deae5fd0, msg=0x7f75deae4830)
    at rpc/virnetserverprogram.c:305
#9  0x00007f75dc582888 in virNetServerProcessMsg (msg=<optimized out>, prog=<optimized out>, client=<optimized out>, srv=0x7f75dead6c70)
    at rpc/virnetserver.c:165
#10 virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x7f75dead6c70) at rpc/virnetserver.c:186
#11 0x00007f75dc4ac425 in virThreadPoolWorker (opaque=opaque@entry=0x7f75dead69a0) at util/virthreadpool.c:144
#12 0x00007f75dc4abea1 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161
#13 0x00007f75d9f9dde3 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f75d98c41ad in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7f75cd3ff700 (LWP 16444)):
#0  0x00007f75d9fa16f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f75dc4ac066 in virCondWait (c=c@entry=0x7f75dead6e50, m=m@entry=0x7f75dead6e28) at util/virthreadpthread.c:117
#2  0x00007f75dc4ac4bb in virThreadPoolWorker (opaque=opaque@entry=0x7f75dea68fc0) at util/virthreadpool.c:103
#3  0x00007f75dc4abea1 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161
#4  0x00007f75d9f9dde3 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f75d98c41ad in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7f75ccbfe700 (LWP 16445)):
#0  0x00007f75d9fa16f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#1  0x00007f75dc4ac066 in virCondWait (c=c@entry=0x7f75dead6e50, m=m@entry=0x7f75dead6e28) at util/virthreadpthread.c:117
#2  0x00007f75dc4ac4bb in virThreadPoolWorker (opaque=opaque@entry=0x7f75dead69a0) at util/virthreadpool.c:103
#3  0x00007f75dc4abea1 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161
#4  0x00007f75d9f9dde3 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f75d98c41ad in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f75cc3fd700 (LWP 16446)):
#0  0x00007f75d9fa16f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f75dc4ac066 in virCondWait (c=c@entry=0x7f75dead6e50, m=m@entry=0x7f75dead6e28) at util/virthreadpthread.c:117
#2  0x00007f75dc4ac4bb in virThreadPoolWorker (opaque=opaque@entry=0x7f75dea68fc0) at util/virthreadpool.c:103
#3  0x00007f75dc4abea1 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161
#4  0x00007f75d9f9dde3 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f75d98c41ad in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7f75cbbfc700 (LWP 16447)):
#0  0x00007f75d9fa16f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f75dc4ac066 in virCondWait (c=c@entry=0x7f75dead6e50, m=m@entry=0x7f75dead6e28) at util/virthreadpthread.c:117
#2  0x00007f75dc4ac4bb in virThreadPoolWorker (opaque=opaque@entry=0x7f75dead69a0) at util/virthreadpool.c:103
#3  0x00007f75dc4abea1 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161
#4  0x00007f75d9f9dde3 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f75d98c41ad in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7f75cb3fb700 (LWP 16448)):
#0  0x00007f75d9fa16f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f75dc4ac066 in virCondWait (c=c@entry=0x7f75dead6ee8, m=m@entry=0x7f75dead6e28) at util/virthreadpthread.c:117
#2  0x00007f75dc4ac4db in virThreadPoolWorker (opaque=opaque@entry=0x7f75dea68fc0) at util/virthreadpool.c:103
#3  0x00007f75dc4abea1 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161
#4  0x00007f75d9f9dde3 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f75d98c41ad in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7f75cabfa700 (LWP 16449)):
#0  0x00007f75d9fa16f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f75dc4ac066 in virCondWait (c=c@entry=0x7f75dead6ee8, m=m@entry=0x7f75dead6e28) at util/virthreadpthread.c:117
---Type <return> to continue, or q <return> to quit---
#2  0x00007f75dc4ac4db in virThreadPoolWorker (opaque=opaque@entry=0x7f75dead69a0) at util/virthreadpool.c:103
#3  0x00007f75dc4abea1 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161
#4  0x00007f75d9f9dde3 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f75d98c41ad in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7f75ca3f9700 (LWP 16450)):
#0  0x00007f75d9fa16f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f75dc4ac066 in virCondWait (c=c@entry=0x7f75dead6ee8, m=m@entry=0x7f75dead6e28) at util/virthreadpthread.c:117
#2  0x00007f75dc4ac4db in virThreadPoolWorker (opaque=opaque@entry=0x7f75dea68fc0) at util/virthreadpool.c:103
#3  0x00007f75dc4abea1 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161
#4  0x00007f75d9f9dde3 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f75d98c41ad in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f75c9bf8700 (LWP 16451)):
#0  0x00007f75d9fa16f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f75dc4ac066 in virCondWait (c=c@entry=0x7f75dead6ee8, m=m@entry=0x7f75dead6e28) at util/virthreadpthread.c:117
#2  0x00007f75dc4ac4db in virThreadPoolWorker (opaque=opaque@entry=0x7f75dea68e10) at util/virthreadpool.c:103
#3  0x00007f75dc4abea1 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161
#4  0x00007f75d9f9dde3 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f75d98c41ad in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f75c93f7700 (LWP 16452)):
#0  0x00007f75d9fa16f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f75dc4ac066 in virCondWait (c=c@entry=0x7f75dead6ee8, m=m@entry=0x7f75dead6e28) at util/virthreadpthread.c:117
#2  0x00007f75dc4ac4db in virThreadPoolWorker (opaque=opaque@entry=0x7f75dea68fc0) at util/virthreadpool.c:103
#3  0x00007f75dc4abea1 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161
#4  0x00007f75d9f9dde3 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f75d98c41ad in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f75dced0880 (LWP 16442)):
#0  0x00007f75d98b9bdd in poll () from /lib64/libc.so.6
#1  0x00007f75dc48149f in poll (__timeout=4999, __nfds=9, __fds=<optimized out>) at /usr/include/bits/poll2.h:46
#2  virEventPollRunOnce () at util/vireventpoll.c:628
---Type <return> to continue, or q <return> to quit---
#3  0x00007f75dc48008d in virEventRunDefaultImpl () at util/virevent.c:273
#4  0x00007f75dc583c1d in virNetServerRun (srv=0x7f75dead6c70) at rpc/virnetserver.c:1112
#5  0x00007f75dcf128dc in main (argc=<optimized out>, argv=<optimized out>) at libvirtd.c:1513




Actual results:
libvirtd crashed

Expected results:
Not crash

Additional info:

Comment 4 Luwen Su 2013-11-28 02:34:40 UTC
Verified with
libvirt-1.1.1-13.el7.x86_64

1.Define , start and destroy a pool that use vHBA

   <pool type='scsi'>
      <name>fc-pool</name>
      <source>
        <adapter type='fc_host'  wwnn='2101001b32a90002' wwpn='2101001b32a90003'/>
      </source>
      <target>
        <path>/dev/disk/by-path</path>
        <permissions>
          <mode>0700</mode>
          <owner>0</owner>
          <group>0</group>
        </permissions>
      </target>
    </pool>

Note:the wwnn,wwpn pair number is valid and have mapped & zoned storage by the admin.

Not crash

2.Define , start and destroy a pool that use HBA

   <pool type='scsi'>
      <name>fc-pool</name>
      <source>
        <adapter type='fc_host'  wwnn='2001001b32a9da4e' wwpn='2001001b32a9da4e'/>
      </source>
      <target>
        <path>/dev/disk/by-path</path>
        <permissions>
          <mode>0700</mode>
          <owner>0</owner>
          <group>0</group>
        </permissions>
      </target>
    </pool>

Not crash

Note: the step 2 is a totally wrong action  , since the wwnn,wwpn number should be vHBAs , not HBAs. The reason why use it here , is just make sure the WRONG behaviour will not crash libvirtd.


Through above , set it VERIFIED

Comment 5 Osier Yang 2013-11-28 04:48:16 UTC
(In reply to time.su from comment #4)
> Verified with
> libvirt-1.1.1-13.el7.x86_64
> 
> 1.Define , start and destroy a pool that use vHBA
> 
>    <pool type='scsi'>
>       <name>fc-pool</name>
>       <source>
>         <adapter type='fc_host'  wwnn='2101001b32a90002'
> wwpn='2101001b32a90003'/>
>       </source>
>       <target>
>         <path>/dev/disk/by-path</path>
>         <permissions>
>           <mode>0700</mode>
>           <owner>0</owner>
>           <group>0</group>
>         </permissions>
>       </target>
>     </pool>
> 
> Note:the wwnn,wwpn pair number is valid and have mapped & zoned storage by
> the admin.
> 
> Not crash
> 
> 2.Define , start and destroy a pool that use HBA
> 
>    <pool type='scsi'>
>       <name>fc-pool</name>
>       <source>
>         <adapter type='fc_host'  wwnn='2001001b32a9da4e'
> wwpn='2001001b32a9da4e'/>
>       </source>
>       <target>
>         <path>/dev/disk/by-path</path>
>         <permissions>
>           <mode>0700</mode>
>           <owner>0</owner>
>           <group>0</group>
>         </permissions>
>       </target>
>     </pool>
> 
> Not crash
> 
> Note: the step 2 is a totally wrong action  , since the wwnn,wwpn number
> should be vHBAs , not HBAs. The reason why use it here , is just make sure
> the WRONG behaviour will not crash libvirtd.
> 

No, it's actually right, we support to define a scsi pool based on a HBA too. I'm not sure if it's caused by the misunderstanding in our previous talk, but hope this clarifies it.

Comment 6 Luwen Su 2013-11-28 09:04:41 UTC
(In reply to Osier Yang from comment #5)
> (In reply to time.su from comment #4)
> > Verified with
> > libvirt-1.1.1-13.el7.x86_64
> > 
> > 1.Define , start and destroy a pool that use vHBA
> > 
> >    <pool type='scsi'>
> >       <name>fc-pool</name>
> >       <source>
> >         <adapter type='fc_host'  wwnn='2101001b32a90002'
> > wwpn='2101001b32a90003'/>
> >       </source>
> >       <target>
> >         <path>/dev/disk/by-path</path>
> >         <permissions>
> >           <mode>0700</mode>
> >           <owner>0</owner>
> >           <group>0</group>
> >         </permissions>
> >       </target>
> >     </pool>
> > 
> > Note:the wwnn,wwpn pair number is valid and have mapped & zoned storage by
> > the admin.
> > 
> > Not crash
> > 
> > 2.Define , start and destroy a pool that use HBA
> > 
> >    <pool type='scsi'>
> >       <name>fc-pool</name>
> >       <source>
> >         <adapter type='fc_host'  wwnn='2001001b32a9da4e'
> > wwpn='2001001b32a9da4e'/>
> >       </source>
> >       <target>
> >         <path>/dev/disk/by-path</path>
> >         <permissions>
> >           <mode>0700</mode>
> >           <owner>0</owner>
> >           <group>0</group>
> >         </permissions>
> >       </target>
> >     </pool>
> > 
> > Not crash
> > 
> > Note: the step 2 is a totally wrong action  , since the wwnn,wwpn number
> > should be vHBAs , not HBAs. The reason why use it here , is just make sure
> > the WRONG behaviour will not crash libvirtd.
> > 
> 
> No, it's actually right, we support to define a scsi pool based on a HBA
> too. I'm not sure if it's caused by the misunderstanding in our previous
> talk, but hope this clarifies it.


Sorry , i must be confused with the nodedev-destroy .Thanks a lot for correct me.

Comment 8 Ludek Smid 2014-06-13 09:30:27 UTC
This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.