Bug 1506287

It appears that when dm devices are stacked that they need to be removed in reverse order.  If you use dmsetup remove_all --force or systematically remove devices using the deferred remove flag in an incorrect order you cause this kernel oops.  If this is expected we can certainly close out this issue.

(In reply to Tony Asleson from comment #1)
> It appears that when dm devices are stacked that they need to be removed in
> reverse order.  If you use dmsetup remove_all --force or systematically
> remove devices using the deferred remove flag in an incorrect order you
> cause this kernel oops.  If this is expected we can certainly close out this
> issue.

Yeah, don't use remove_all --force.  Using --force is a HUGE hammer.  Your testing needs to be way more careful to teardown more systematically.

As for using deferred removal.  That _should_ be OK.  Because it doesn't actually remove the device until it is no longer used.

So I'm not sure why you'd be having problems with it.

(In reply to Mike Snitzer from comment #2)
> As for using deferred removal.  That _should_ be OK.  Because it doesn't
> actually remove the device until it is no longer used.
> 
> So I'm not sure why you'd be having problems with it.

Well maybe there is a bug when using the deferred flag... when I don't use the flag I get an error that device is busy, but I don't oops the kernel.

Basically I just iterate over all the dm devices doing a remove with the deferred flag, so in theory it should result in all the devices getting removed eventually, but doing so I can cause a kernel oops pretty easily.  If I repeatedly loop over the device list without the deferred flag, I can get all the devices to get removed without causing a kernel oops while getting the device in use errors.

I discovered that the unit tests were being massively threaded by default, thus we had concurrent access to the device mapper ioctl interface.  I would still assume that in this use case that the dm code would not end up dereferencing a NULL pointer.  Hopefully this will help in root cause determination.

The kernel trace suggest - there was already a trouble reported by thin-pool targets.


remove --force  is 'very nasty' last-defense logic implement to help with cases the device is holding some resources but cannot be closed - so with --force at least 'error' target is put as replacement.

However this 'replacement' is NOT support with more complex target like thin-pool  as some operation inside this complex target do support only 'clean' target removal not such brute force trying.

So I assume there can be code path where thin-pool is put into an untested territory where all 'reads/write'  fail in  preresume case.

The logic behind kernel 'preresume' somewhat changed it's original meaning of simple validation of user's passed state into a very complex 'metadata' parsing stage and at it's known 'postponned' issue. 

There is problem that such preresume/resume failure is typically unsolvable by any user-land app as happens after commit point...

BTW. what does the dmsetup --force flag really do?

My tests show that it does nothing (dmsetup remove --force or remove_all --force don't remove an open device). And looking at the kernel source, I don't see any code that tests for that flag at all in dev_remove or remove_all.

dm_hash_remove_all will perform forced remove only if the first argument is false, but it is called with the first false argument only from dm_hash_exit and that is called from dm_interface_init and dm_interface_exit - that is module initialization and cleanup and it can't be triggered by userspace ioctls.

I.e. there seems to be no way how userspace could trigger forced removal of a device mapper device.

BTW. could it be that your test scripts do "rmmod -f"? That could cause crashes and the scripts shouldn't do that.

(In reply to Mikulas Patocka from comment #6)
> BTW. what does the dmsetup --force flag really do?
> 
> My tests show that it does nothing (dmsetup remove --force or remove_all
> --force don't remove an open device). And looking at the kernel source, I
> don't see any code that tests for that flag at all in dev_remove or
> remove_all.
> 


Just to answer this specifically -

'dmsetup remove --force'   

tries to 'remove'  a device - however if device is found to be in-use (opened) - it cannot be dropped from table - but resource associated with a device can be usually dropped -

so i.e. if linear device holds a reference to  /dev/sdX device - it can be dropped by replacing target with 'error' segment of the equal size.

The clear purpose is - any further I/O gets error (possibly making it even possible to actually close such device -  of course this may from time to time hit a bug on the road - but i.e.   ext4 or XFS handles such case well enough these days.

But of course - whever user is using --force - he is doing something seriously unwanted and should consider twice whether such risk is worth...

Zdenek Kabeláč: replacing a device with error target shouldn't crash - so the advices to not use --force are not correct and we should find the bug and fix it.