Bug 812927

Summary: lvm hung after the thin pool is full
Product: [Fedora] Fedora Reporter: Xiaowei Li <xiaoli>
Component: lvm2Assignee: Zdenek Kabelac <zkabelac>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: 23CC: agk, bmarzins, bmr, dustymabe, dwysocha, heinzm, jonathan, jsynacek, lvm-team, msnitzer, prajnoha, prockai, qcai, zkabelac
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-12-07 09:58:56 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
dmesg output none

Description Xiaowei Li 2012-04-16 15:19:55 UTC
Description of problem:
lvm hung after the thin pool is full

Version-Release number of selected component (if applicable):

How reproducible:
always

Steps to Reproduce:
1. pvcreate /dev/vdb
2. vgcreate vg /dev/vdb
3. lvcreate -L 100M -T vg/pool
4. lvcreate -V 120M -T -n tp vg/pool
5. dd if=/dev/zero of=/dev/vg/tp bs=1M count=120 & 
6. >>>the thin pool is full<< sleep 30
7. lvs 
8. lvextend -L 200M vg/pool
  
Actual results:
both lvs and lvextend hung after the thin pool is full

Expected results:
should not suspend the lvm function

Additional info:

Comment 1 Xiaowei Li 2012-04-16 15:30:50 UTC
after rebooting the box
[root@laker ~]# lvs
  LV      VG       Attr     LSize   Pool Origin Data%  Move Log Copy%  Convert
  pool    vg       twi-a-tz 100.00m             100.00                        
  tp      vg       Vwi-a-tz 120.00m pool          0.00   

the thin pool is 100% full but the thin disk is 0%. This is strange.

I think should not block the read from the full thin pool.

what's the lvm designed behaviour if the thin pool is full?

Comment 2 Alasdair Kergon 2012-04-16 15:34:27 UTC
Eventually the device will become read-only at the point it fills up, with any further writes lost.  (A configurable 'queue' option might also be possible, similar to multipath's 'queue if no path'.)

But kernel patches to handle these situations have not yet been written.

Did the background 'dd' exit or hang?

Comment 3 Xiaowei Li 2012-04-16 16:52:19 UTC
(In reply to comment #2)
> Eventually the device will become read-only at the point it fills up, with any
> further writes lost.  (A configurable 'queue' option might also be possible,
> similar to multipath's 'queue if no path'.)
> 
> But kernel patches to handle these situations have not yet been written.
> 
> Did the background 'dd' exit or hang?

Thanks for the clarification.

it hang. and the lvm relate commands hung if invoking them.

Comment 4 Fedora End Of Life 2013-07-04 06:37:45 UTC
This message is a reminder that Fedora 17 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 17. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '17'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 17's end of life.

Bug Reporter:  Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 17 is end of life. If you 
would still like  to see this bug fixed and are able to reproduce it 
against a later version  of Fedora, you are encouraged  change the 
'version' to a later Fedora version prior to Fedora 17's end of life.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 5 Fedora End Of Life 2013-08-01 18:14:09 UTC
Fedora 17 changed to end-of-life (EOL) status on 2013-07-30. Fedora 17 is 
no longer maintained, which means that it will not receive any further 
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of 
Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.

Comment 6 Alasdair Kergon 2013-10-04 11:29:13 UTC
This should NOT have been closed.

Comment 7 Jan Synacek 2013-10-04 11:53:36 UTC
Created attachment 807600 [details]
dmesg output

I ran into this today on the latest rawhide (f21). See the attached dmesg output, timestamp 557-ish.

I have a 1GB thin pool and a 10GB thin lv. Whem attempting to write about 1.4GB to the thin lv (using midnight commander or dd, it doesn't matter), the process that does the writing hangs. Every other process that attempts to access the lvm hangs as well.

The thin lv is formatted as ext4.

# rpm -q lvm2
lvm2-2.02.102-1.fc21.x86_64

# uname -a
Linux rawhide-virt 3.12.0-0.rc3.git1.1.fc21.x86_64 #1 SMP Tue Oct 1 13:34:25 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

I'm running the system in a virtual machine, if that makes a difference.

Comment 8 Jan Synacek 2013-10-04 11:55:43 UTC
# vgs
  VG   #PV #LV #SN Attr   VSize VFree
  vg1    3   4   0 wz--n- 2.84g 1.59g

# lvs
  LV       VG   Attr       LSize  Pool     Origin Data%  Move Log Cpy%Sync Convert
  lv1      vg1  -wi-a----- 96.00m                                                 
  lv2      vg1  -wi-a----- 96.00m                                                 
  thin1    vg1  Vwi-aotz-- 10.00g thinpool         10.00                          
  thinpool vg1  twi-a-tz--  1.00g                 100.00

# vgdisplay 
  --- Volume group ---
  VG Name               vg1
  System ID             
  Format                lvm2
  Metadata Areas        3
  Metadata Sequence No  84
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                4
  Open LV               1
  Max PV                0
  Cur PV                3
  Act PV                3
  VG Size               2.84 GiB
  PE Size               32.00 MiB
  Total PE              91
  Alloc PE / Size       40 / 1.25 GiB
  Free  PE / Size       51 / 1.59 GiB
  VG UUID               5JZ70v-xre0-ZyJX-yZVO-ecKd-anBg-QVQKb9

# lvdisplay 
  --- Logical volume ---
  LV Path                /dev/vg1/lv1
  LV Name                lv1
  VG Name                vg1
  LV UUID                mTLbdB-Nc6t-IaMJ-LWuY-JloP-cfsh-I2IzSL
  LV Write Access        read/write
  LV Creation host, time rawhide-virt, 2013-09-11 10:57:15 +0200
  LV Status              available
  # open                 0
  LV Size                96.00 MiB
  Current LE             3
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0
   
  --- Logical volume ---
  LV Path                /dev/vg1/lv2
  LV Name                lv2
  VG Name                vg1
  LV UUID                wBAUYw-pbon-4bzD-LtHY-mopn-miR7-YdE5uJ
  LV Write Access        read/write
  LV Creation host, time rawhide-virt, 2013-09-11 10:57:31 +0200
  LV Status              available
  # open                 0
  LV Size                96.00 MiB
  Current LE             3
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:2
   
  --- Logical volume ---
  LV Name                thinpool
  VG Name                vg1
  LV UUID                3L6J9P-sDuh-2xOv-Jbsi-1vcR-Rc3K-4BRjvB
  LV Write Access        read/write
  LV Creation host, time rawhide-virt, 2013-10-04 10:29:57 +0200
  LV Pool transaction ID 3
  LV Pool metadata       thinpool_tmeta
  LV Pool data           thinpool_tdata
  LV Pool chunk size     256.00 KiB
  LV Zero new blocks     yes
  LV Status              available
  # open                 0
  LV Size                1.00 GiB
  Allocated pool data    100.00%
  Allocated metadata     0.49%
  Current LE             32
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:5
   
  --- Logical volume ---
  LV Path                /dev/vg1/thin1
  LV Name                thin1
  VG Name                vg1
  LV UUID                VCtLj6-Z84W-XoU1-CAUt-0BFw-ldDV-YVOBGY
  LV Write Access        read/write
  LV Creation host, time rawhide-virt, 2013-10-04 10:50:50 +0200
  LV Pool name           thinpool
  LV Status              available
  # open                 1
  LV Size                10.00 GiB
  Mapped size            10.00%
  Current LE             320
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:6

Comment 9 Jan Synacek 2013-10-04 12:08:08 UTC
I just found out that extending the thin pool so it's big enough releases all the locks.

Comment 11 Jan Kurik 2015-07-15 15:09:57 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 23 development cycle.
Changing version to '23'.

(As we did not run this process for some time, it could affect also pre-Fedora 23 development
cycle bugs. We are very sorry. It will help us with cleanup during Fedora 23 End Of Life. Thank you.)

More information and reason for this action is here:
https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora23

Comment 12 Zdenek Kabelac 2015-12-07 09:58:56 UTC
Marking this bug as closed with current release - new problems should be reported as new bug.

lvm2 now properly should resize thin-pool (when configured via lvm.conf).