Bug 1027680 - Destroy a NPIV pool which lack of adapter's parent lead to libvirtd crash
Destroy a NPIV pool which lack of adapter's parent lead to libvirtd crash
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt (Show other bugs)
7.0
Unspecified Unspecified
medium Severity medium
: rc
: ---
Assigned To: John Ferlan
Virtualization Bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-11-07 04:35 EST by Luwen Su
Modified: 2014-06-17 20:58 EDT (History)
5 users (show)

See Also:
Fixed In Version: libvirt-1.1.1-13.el7
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-06-13 05:30:27 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Luwen Su 2013-11-07 04:35:10 EST
Description of problem:
Destroy a pool which lack of elemet parent lead to libvirtd crash

Version-Release number of selected component (if applicable):
libvirt-1.1.1-11.el7.x86_64
kernel-3.10.0-41.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Prepare the pool's xml
    <pool type='scsi'>
      <name>test</name>
      <source>
        <adapter type='fc_host'   wwnn='20000024ff370144'wwpn='20000024ff370144'/>
      </source>
      <target>
        <path>/dev/disk/by-path/</path>
        <permissions>
          <mode>0755</mode>
          <owner>-1</owner>
          <group>-1</group>
        </permissions>
      </target>
    </pool>


the wwnn and wwpn pointed to a vport in my host's 
 #virsh nodedev-dumpxml scsi_host1
<device>
  <name>scsi_host1</name>
  <path>/sys/devices/pci0000:00/0000:00:09.0/0000:18:00.0/host1</path>
  <parent>pci_0000_18_00_0</parent>
  <capability type='scsi_host'>
    <host>1</host>
    <capability type='fc_host'>
      <wwnn>20000024ff370144</wwnn>
      <wwpn>20000024ff370144</wwpn>
      <fabric_wwn>2001000dec9877c1</fabric_wwn>
    </capability>
    <capability type='vport_ops'>
      <max_vports>254</max_vports>
      <vports>0</vports>
    </capability>
  </capability>
</device>

2.do
#virsh pool-define pool.xml
#virsh pool-start test
#virsh pool-destroy test
error: Failed to destroy pool test
error: End of file while reading data: Input/output error
error: One or more references were leaked after disconnect from the hypervisor
error: Failed to reconnect to the hypervisor


3.Gdb's backtrace

(gdb)thread apply all backtrace
Thread 11 (Thread 0x7f75cdc00700 (LWP 16443)):
#0  getHostNumber (adapter_name=adapter_name@entry=0x0, result=result@entry=0x7f75cdbffacc) at storage/storage_backend_scsi.c:579
#1  0x00007f75c87bd43f in deleteVport (adapter=...) at storage/storage_backend_scsi.c:671
#2  virStorageBackendSCSIStopPool (conn=<optimized out>, pool=<optimized out>) at storage/storage_backend_scsi.c:755
#3  0x00007f75c87b11e5 in storagePoolDestroy (obj=0x7f75a8000c50) at storage/storage_driver.c:874
#4  0x00007f75dc53b8a6 in virStoragePoolDestroy (pool=pool@entry=0x7f75a8000c50) at libvirt.c:13859
#5  0x00007f75dcf2cfa8 in remoteDispatchStoragePoolDestroy (server=<optimized out>, msg=<optimized out>, args=<optimized out>, rerr=0x7f75cdbffc90, 
    client=0x7f75deae5fd0) at remote_dispatch.h:12412
#6  remoteDispatchStoragePoolDestroyHelper (server=<optimized out>, client=0x7f75deae5fd0, msg=<optimized out>, rerr=0x7f75cdbffc90, args=<optimized out>, 
    ret=<optimized out>) at remote_dispatch.h:12390
#7  0x00007f75dc587a4a in virNetServerProgramDispatchCall (msg=0x7f75deae4830, client=0x7f75deae5fd0, server=0x7f75dead6c70, prog=0x7f75deae0570)
    at rpc/virnetserverprogram.c:435
#8  virNetServerProgramDispatch (prog=0x7f75deae0570, server=server@entry=0x7f75dead6c70, client=0x7f75deae5fd0, msg=0x7f75deae4830)
    at rpc/virnetserverprogram.c:305
#9  0x00007f75dc582888 in virNetServerProcessMsg (msg=<optimized out>, prog=<optimized out>, client=<optimized out>, srv=0x7f75dead6c70)
    at rpc/virnetserver.c:165
#10 virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x7f75dead6c70) at rpc/virnetserver.c:186
#11 0x00007f75dc4ac425 in virThreadPoolWorker (opaque=opaque@entry=0x7f75dead69a0) at util/virthreadpool.c:144
#12 0x00007f75dc4abea1 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161
#13 0x00007f75d9f9dde3 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f75d98c41ad in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7f75cd3ff700 (LWP 16444)):
#0  0x00007f75d9fa16f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f75dc4ac066 in virCondWait (c=c@entry=0x7f75dead6e50, m=m@entry=0x7f75dead6e28) at util/virthreadpthread.c:117
#2  0x00007f75dc4ac4bb in virThreadPoolWorker (opaque=opaque@entry=0x7f75dea68fc0) at util/virthreadpool.c:103
#3  0x00007f75dc4abea1 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161
#4  0x00007f75d9f9dde3 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f75d98c41ad in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7f75ccbfe700 (LWP 16445)):
#0  0x00007f75d9fa16f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#1  0x00007f75dc4ac066 in virCondWait (c=c@entry=0x7f75dead6e50, m=m@entry=0x7f75dead6e28) at util/virthreadpthread.c:117
#2  0x00007f75dc4ac4bb in virThreadPoolWorker (opaque=opaque@entry=0x7f75dead69a0) at util/virthreadpool.c:103
#3  0x00007f75dc4abea1 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161
#4  0x00007f75d9f9dde3 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f75d98c41ad in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f75cc3fd700 (LWP 16446)):
#0  0x00007f75d9fa16f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f75dc4ac066 in virCondWait (c=c@entry=0x7f75dead6e50, m=m@entry=0x7f75dead6e28) at util/virthreadpthread.c:117
#2  0x00007f75dc4ac4bb in virThreadPoolWorker (opaque=opaque@entry=0x7f75dea68fc0) at util/virthreadpool.c:103
#3  0x00007f75dc4abea1 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161
#4  0x00007f75d9f9dde3 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f75d98c41ad in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7f75cbbfc700 (LWP 16447)):
#0  0x00007f75d9fa16f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f75dc4ac066 in virCondWait (c=c@entry=0x7f75dead6e50, m=m@entry=0x7f75dead6e28) at util/virthreadpthread.c:117
#2  0x00007f75dc4ac4bb in virThreadPoolWorker (opaque=opaque@entry=0x7f75dead69a0) at util/virthreadpool.c:103
#3  0x00007f75dc4abea1 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161
#4  0x00007f75d9f9dde3 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f75d98c41ad in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7f75cb3fb700 (LWP 16448)):
#0  0x00007f75d9fa16f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f75dc4ac066 in virCondWait (c=c@entry=0x7f75dead6ee8, m=m@entry=0x7f75dead6e28) at util/virthreadpthread.c:117
#2  0x00007f75dc4ac4db in virThreadPoolWorker (opaque=opaque@entry=0x7f75dea68fc0) at util/virthreadpool.c:103
#3  0x00007f75dc4abea1 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161
#4  0x00007f75d9f9dde3 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f75d98c41ad in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7f75cabfa700 (LWP 16449)):
#0  0x00007f75d9fa16f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f75dc4ac066 in virCondWait (c=c@entry=0x7f75dead6ee8, m=m@entry=0x7f75dead6e28) at util/virthreadpthread.c:117
---Type <return> to continue, or q <return> to quit---
#2  0x00007f75dc4ac4db in virThreadPoolWorker (opaque=opaque@entry=0x7f75dead69a0) at util/virthreadpool.c:103
#3  0x00007f75dc4abea1 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161
#4  0x00007f75d9f9dde3 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f75d98c41ad in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7f75ca3f9700 (LWP 16450)):
#0  0x00007f75d9fa16f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f75dc4ac066 in virCondWait (c=c@entry=0x7f75dead6ee8, m=m@entry=0x7f75dead6e28) at util/virthreadpthread.c:117
#2  0x00007f75dc4ac4db in virThreadPoolWorker (opaque=opaque@entry=0x7f75dea68fc0) at util/virthreadpool.c:103
#3  0x00007f75dc4abea1 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161
#4  0x00007f75d9f9dde3 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f75d98c41ad in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f75c9bf8700 (LWP 16451)):
#0  0x00007f75d9fa16f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f75dc4ac066 in virCondWait (c=c@entry=0x7f75dead6ee8, m=m@entry=0x7f75dead6e28) at util/virthreadpthread.c:117
#2  0x00007f75dc4ac4db in virThreadPoolWorker (opaque=opaque@entry=0x7f75dea68e10) at util/virthreadpool.c:103
#3  0x00007f75dc4abea1 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161
#4  0x00007f75d9f9dde3 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f75d98c41ad in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f75c93f7700 (LWP 16452)):
#0  0x00007f75d9fa16f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f75dc4ac066 in virCondWait (c=c@entry=0x7f75dead6ee8, m=m@entry=0x7f75dead6e28) at util/virthreadpthread.c:117
#2  0x00007f75dc4ac4db in virThreadPoolWorker (opaque=opaque@entry=0x7f75dea68fc0) at util/virthreadpool.c:103
#3  0x00007f75dc4abea1 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161
#4  0x00007f75d9f9dde3 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f75d98c41ad in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f75dced0880 (LWP 16442)):
#0  0x00007f75d98b9bdd in poll () from /lib64/libc.so.6
#1  0x00007f75dc48149f in poll (__timeout=4999, __nfds=9, __fds=<optimized out>) at /usr/include/bits/poll2.h:46
#2  virEventPollRunOnce () at util/vireventpoll.c:628
---Type <return> to continue, or q <return> to quit---
#3  0x00007f75dc48008d in virEventRunDefaultImpl () at util/virevent.c:273
#4  0x00007f75dc583c1d in virNetServerRun (srv=0x7f75dead6c70) at rpc/virnetserver.c:1112
#5  0x00007f75dcf128dc in main (argc=<optimized out>, argv=<optimized out>) at libvirtd.c:1513




Actual results:
libvirtd crashed

Expected results:
Not crash

Additional info:
Comment 4 Luwen Su 2013-11-27 21:34:40 EST
Verified with
libvirt-1.1.1-13.el7.x86_64

1.Define , start and destroy a pool that use vHBA

   <pool type='scsi'>
      <name>fc-pool</name>
      <source>
        <adapter type='fc_host'  wwnn='2101001b32a90002' wwpn='2101001b32a90003'/>
      </source>
      <target>
        <path>/dev/disk/by-path</path>
        <permissions>
          <mode>0700</mode>
          <owner>0</owner>
          <group>0</group>
        </permissions>
      </target>
    </pool>

Note:the wwnn,wwpn pair number is valid and have mapped & zoned storage by the admin.

Not crash

2.Define , start and destroy a pool that use HBA

   <pool type='scsi'>
      <name>fc-pool</name>
      <source>
        <adapter type='fc_host'  wwnn='2001001b32a9da4e' wwpn='2001001b32a9da4e'/>
      </source>
      <target>
        <path>/dev/disk/by-path</path>
        <permissions>
          <mode>0700</mode>
          <owner>0</owner>
          <group>0</group>
        </permissions>
      </target>
    </pool>

Not crash

Note: the step 2 is a totally wrong action  , since the wwnn,wwpn number should be vHBAs , not HBAs. The reason why use it here , is just make sure the WRONG behaviour will not crash libvirtd.


Through above , set it VERIFIED
Comment 5 Osier Yang 2013-11-27 23:48:16 EST
(In reply to time.su from comment #4)
> Verified with
> libvirt-1.1.1-13.el7.x86_64
> 
> 1.Define , start and destroy a pool that use vHBA
> 
>    <pool type='scsi'>
>       <name>fc-pool</name>
>       <source>
>         <adapter type='fc_host'  wwnn='2101001b32a90002'
> wwpn='2101001b32a90003'/>
>       </source>
>       <target>
>         <path>/dev/disk/by-path</path>
>         <permissions>
>           <mode>0700</mode>
>           <owner>0</owner>
>           <group>0</group>
>         </permissions>
>       </target>
>     </pool>
> 
> Note:the wwnn,wwpn pair number is valid and have mapped & zoned storage by
> the admin.
> 
> Not crash
> 
> 2.Define , start and destroy a pool that use HBA
> 
>    <pool type='scsi'>
>       <name>fc-pool</name>
>       <source>
>         <adapter type='fc_host'  wwnn='2001001b32a9da4e'
> wwpn='2001001b32a9da4e'/>
>       </source>
>       <target>
>         <path>/dev/disk/by-path</path>
>         <permissions>
>           <mode>0700</mode>
>           <owner>0</owner>
>           <group>0</group>
>         </permissions>
>       </target>
>     </pool>
> 
> Not crash
> 
> Note: the step 2 is a totally wrong action  , since the wwnn,wwpn number
> should be vHBAs , not HBAs. The reason why use it here , is just make sure
> the WRONG behaviour will not crash libvirtd.
> 

No, it's actually right, we support to define a scsi pool based on a HBA too. I'm not sure if it's caused by the misunderstanding in our previous talk, but hope this clarifies it.
Comment 6 Luwen Su 2013-11-28 04:04:41 EST
(In reply to Osier Yang from comment #5)
> (In reply to time.su from comment #4)
> > Verified with
> > libvirt-1.1.1-13.el7.x86_64
> > 
> > 1.Define , start and destroy a pool that use vHBA
> > 
> >    <pool type='scsi'>
> >       <name>fc-pool</name>
> >       <source>
> >         <adapter type='fc_host'  wwnn='2101001b32a90002'
> > wwpn='2101001b32a90003'/>
> >       </source>
> >       <target>
> >         <path>/dev/disk/by-path</path>
> >         <permissions>
> >           <mode>0700</mode>
> >           <owner>0</owner>
> >           <group>0</group>
> >         </permissions>
> >       </target>
> >     </pool>
> > 
> > Note:the wwnn,wwpn pair number is valid and have mapped & zoned storage by
> > the admin.
> > 
> > Not crash
> > 
> > 2.Define , start and destroy a pool that use HBA
> > 
> >    <pool type='scsi'>
> >       <name>fc-pool</name>
> >       <source>
> >         <adapter type='fc_host'  wwnn='2001001b32a9da4e'
> > wwpn='2001001b32a9da4e'/>
> >       </source>
> >       <target>
> >         <path>/dev/disk/by-path</path>
> >         <permissions>
> >           <mode>0700</mode>
> >           <owner>0</owner>
> >           <group>0</group>
> >         </permissions>
> >       </target>
> >     </pool>
> > 
> > Not crash
> > 
> > Note: the step 2 is a totally wrong action  , since the wwnn,wwpn number
> > should be vHBAs , not HBAs. The reason why use it here , is just make sure
> > the WRONG behaviour will not crash libvirtd.
> > 
> 
> No, it's actually right, we support to define a scsi pool based on a HBA
> too. I'm not sure if it's caused by the misunderstanding in our previous
> talk, but hope this clarifies it.


Sorry , i must be confused with the nodedev-destroy .Thanks a lot for correct me.
Comment 8 Ludek Smid 2014-06-13 05:30:27 EDT
This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.

Note You need to log in before you can comment on or make changes to this bug.