Bug 1664652
| Summary: | `ssm list` command delays in detecting the correct free space from newly grown size of the xfs filesystem. | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Nitin U. Yewale <nyewale> | ||||
| Component: | system-storage-manager | Assignee: | Lukáš Czerner <lczerner> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Boyang Xue <bxue> | ||||
| Severity: | medium | Docs Contact: | |||||
| Priority: | medium | ||||||
| Version: | 7.6 | CC: | agk, lczerner, nyewale, prajnoha, xzhou, zkabelac | ||||
| Target Milestone: | rc | ||||||
| Target Release: | --- | ||||||
| Hardware: | All | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2019-08-06 12:55:09 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
|
Description
Nitin U. Yewale
2019-01-09 12:06:10 UTC
We could see that `ssm list`
runs the following command in the background to read the fs size and fs free size information
# xfs_db -r -c sb -c print <device-name>
It has been observed that xfs_db database is getting updated after 30 seconds (because of below parameter) after running xfs_growfs command.
So, in order to get the correct values in `ssm list` it should be run atleast after 30 seconds after running the `xfs_growfs` command.
The parameter `/proc/sys/fs/xfs/xfssyncd_centisecs` governs the sync duration.
We could reduce to minimum recommended of 10 seconds though and the file system 'Free size' gets updated in the ssm command bit early.
# echo 100 > /proc/sys/fs/xfs/xfssyncd_centisecs
We tested this and correct values get reflected in `ssm list` after 10 seconds.
Created attachment 1519476 [details]
Patch to fix the delay in detecting the correct free space of xfs file system everytime
Attached patch helps detect the free space correctly everytime.
Test result after the fix. Please see (./bin/ssm.local list volumes) command which is detecting the correct free space.
`ssm list volumes` command is from unfixed package.
Check the size of file system /test1
[root@stest ssm]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rhel_vm252--140-root 79G 8.7G 70G 12% /
devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 1.9G 9.1M 1.9G 1% /run
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/sda1 2.0G 199M 1.8G 10% /boot
tmpfs 379M 16K 379M 1% /run/user/42
tmpfs 379M 0 379M 0% /run/user/0
/dev/mapper/vg3-lv1 2.5G 33M 2.5G 2% /test1
Grow this file system
[root@stest ssm]# ssm resize -s +400M /dev/vg3/lv1
Size of logical volume vg3/lv1 changed from 2.46 GiB (631 extents) to <2.86 GiB (731 extents).
Logical volume vg3/lv1 successfully resized.
meta-data=/dev/mapper/vg3-lv1 isize=512 agcount=10, agsize=65536 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0 spinodes=0
data = bsize=4096 blocks=646144, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
data blocks changed from 646144 to 748544
[root@stest ssm]# ssm list volumes <------------- Fails to detect the correct size
---------------------------------------------------------------------------------------------------
Volume Pool Volume size FS FS size Free Type Mount point
---------------------------------------------------------------------------------------------------
/dev/rhel_vm252-140/root rhel_vm252-140 78.12 GB xfs 78.09 GB 69.46 GB linear /
/dev/rhel_vm252-140/swap rhel_vm252-140 2.00 GB linear
/dev/vg2/lv22 vg2 2.95 GB xfs 2.94 GB 2.91 GB linear
/dev/vg3/lv1 vg3 2.86 GB xfs 2.46 GB 2.42 GB linear /test1 <----
/dev/vg4/lv4 vg4 1.78 GB xfs 1.77 GB 1.77 GB linear
/dev/sda1 2.00 GB xfs 1.99 GB 1.80 GB part /boot
---------------------------------------------------------------------------------------------------
[root@stest ssm]# ./bin/ssm.local list volumes <--------- With fix, we detect the correct size
---------------------------------------------------------------------------------------------------
Volume Pool Volume size FS FS size Free Type Mount point
---------------------------------------------------------------------------------------------------
/dev/rhel_vm252-140/root rhel_vm252-140 78.12 GB xfs 78.09 TB 69.46 TB linear /
/dev/rhel_vm252-140/swap rhel_vm252-140 2.00 GB linear
/dev/vg2/lv22 vg2 2.95 GB xfs 2.94 GB 2.91 GB linear
/dev/vg3/lv1 vg3 2.86 GB xfs 2.85 TB 2.81 TB linear /test1 <----
/dev/vg4/lv4 vg4 1.78 GB xfs 1.77 GB 1.77 GB linear
/dev/sda1 2.00 GB xfs 1.99 TB 1.80 TB part /boot
---------------------------------------------------------------------------------------------------
[root@stest ssm]# ssm list volumes <---------------------------- This still fails to detect the correct size
---------------------------------------------------------------------------------------------------
Volume Pool Volume size FS FS size Free Type Mount point
---------------------------------------------------------------------------------------------------
/dev/rhel_vm252-140/root rhel_vm252-140 78.12 GB xfs 78.09 GB 69.46 GB linear /
/dev/rhel_vm252-140/swap rhel_vm252-140 2.00 GB linear
/dev/vg2/lv22 vg2 2.95 GB xfs 2.94 GB 2.91 GB linear
/dev/vg3/lv1 vg3 2.86 GB xfs 2.46 GB 2.42 GB linear /test1 <------
/dev/vg4/lv4 vg4 1.78 GB xfs 1.77 GB 1.77 GB linear
/dev/sda1 2.00 GB xfs 1.99 GB 1.80 GB part /boot
---------------------------------------------------------------------------------------------------
[root@stest ssm]#
Thanks for finding the issue and writing the patch, it looks good. I will make a test case for it and try to find if there is a way to tell xfs_db to update the info after ssm resize - and if that is a good idea at all. BTW, this issue seems to affect other distros too, confirmed at current RHEL 7, 8, Fedora 28 and Archlinux. Thank you Jan. We see that the parameter `/proc/sys/fs/xfs/xfssyncd_centisecs` governs how quickly xfs_db gets updated. By default its 30 seconds and could be reduced to 10 seconds at minimum. Just to add. Patch has been created in the upstream git repo. On second thought (and after checking with upstream XFS), xfs_db is generally not the tool to use with a live, mounted XFS at all. With that in mind, there is no need for changing the parameter - xfs_db skips caching, etc., and on a mounted filesystem, it is almost guaranteed to report stale data. So your patch makes the right thing to ask kernel and not xfs_db. I will just add a comment to the code, why it is there the mount check. Hello Jan,
It seems that running os.statvfs on mounted filesystem and running xfs_db on unmounted device gives correct result. As we were running xfs_db on devices, so retained that behavior. Please let me know if I am missing something.
[root@stest tests]# df -h |grep lv
/dev/mapper/vg3-lv1 2.9G 1.6G 1.4G 54% /test1
[root@stest tests]# lvs -a -o +devices |grep vg
lv22 vg2 -wi-a----- 2.95g /dev/sda3(0)
lv1 vg3 -wi-ao---- <2.86g /dev/sdb1(0)
lv4 vg4 -wi-a----- 1.78g /dev/sdb2(0)
--------------------------------------------------------------------------------------------------------------------
================== /dev/vg4/lv4 =================
1968934912
1968934912
============ /test1 ============================
posix.statvfs_result(f_bsize=4096, f_frsize=4096, f_blocks=745984, f_bfree=475540, f_bavail=475540, f_files=1497088, f_ffree=1497084, f_favail=1497084, f_flag=4096, f_namemax=255)
3055550464
1947811840 <-------------------------------------------------------
============ /dev/vg3/lv1 ======================
posix.statvfs_result(f_bsize=4096, f_frsize=4096, f_blocks=480697, f_bfree=480697, f_bavail=480697, f_files=480697, f_ffree=480247, f_favail=480247, f_flag=2, f_namemax=255)
1968934912
1968934912
--------------------------------------------------------------------------------------------------------------------
# dd if=/dev/zero of=/test1/disk2.img bs=1M count=500
500+0 records in
500+0 records out
524288000 bytes (524 MB) copied, 0.293449 s, 1.8 GB/s
--------------------------------------------------------------------------------------------------------------------
[root@stest tests]# python mount_test2.py
================== /dev/vg4/lv4 =================
1968934912
1968934912
============ /test1 ============================
posix.statvfs_result(f_bsize=4096, f_frsize=4096, f_blocks=745984, f_bfree=347540, f_bavail=347540, f_files=1497088, f_ffree=1497083, f_favail=1497083, f_flag=4096, f_namemax=255)
3055550464
1423523840 <-------------------------------------------------------
============ /dev/vg3/lv1 ======================
posix.statvfs_result(f_bsize=4096, f_frsize=4096, f_blocks=480697, f_bfree=480697, f_bavail=480697, f_files=480697, f_ffree=480247, f_favail=480247, f_flag=2, f_namemax=255)
1968934912
1968934912
[root@stest tests]#
# cat mount_test2.py
import os
print ("================== /dev/vg4/lv4 =================")
stat = os.statvfs('/dev/vg4/lv4')
total = stat.f_blocks*stat.f_bsize
free = stat.f_bfree*stat.f_bsize
print(total)
print(free)
print ("============ /test1 ============================")
stat = os.statvfs('/test1')
print(stat)
total = stat.f_blocks*stat.f_bsize
free = stat.f_bfree*stat.f_bsize
print(total)
print(free)
print ("============ /dev/vg3/lv1 ======================")
stat = os.statvfs('/dev/vg3/lv1')
print(stat)
total = stat.f_blocks*stat.f_bsize
free = stat.f_bfree*stat.f_bsize
print(total)
print(free)
Thank you,
Nitin Yewale
[root@stest tests]# df -h |grep lv /dev/mapper/vg3-lv1 2.9G 1.6G 1.4G 54% /test1 This is after running dd command. Is there a reason for keeping this bug private? If possible, I would make it public, so I can refer to it in the upstream. I have the patch prepared for a review but need to know if I can refer to this Bugzilla issue publicly or not. TEST PASS.
Reproduced with ssm version 0.4-8.el7
---
[root@host-8-242-86 tests1minutetip_bringup]# ssm create -s 1G --fs xfs /dev/loop0 && lvs && ssm list fs && lvextend -L +1G lvm_pool/lvol001 && lvs && mount /dev/lvm_pool/lvol001 /media && xfs_growfs /media && ssm list fs && sleep 30s && ssm list fs
Volume group "lvm_pool" successfully created
Logical volume "lvol001" created.
meta-data=/dev/lvm_pool/lvol001 isize=512 agcount=4, agsize=65536 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0, sparse=0
data = bsize=4096 blocks=262144, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
lvol001 lvm_pool -wi-a----- 1.00g
----------------------------------------------------------------------------------------------
Volume Pool Volume size FS FS size Free Type Mount point
----------------------------------------------------------------------------------------------
/dev/lvm_pool/lvol001 lvm_pool 1.00 GB xfs 1014.00 MB 1013.86 MB linear
/dev/vda1 20.00 GB xfs 19.99 GB 18.29 GB part /
----------------------------------------------------------------------------------------------
Size of logical volume lvm_pool/lvol001 changed from 1.00 GiB (256 extents) to 2.00 GiB (512 extents).
Logical volume lvm_pool/lvol001 successfully resized.
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
lvol001 lvm_pool -wi-a----- 2.00g
meta-data=/dev/mapper/lvm_pool-lvol001 isize=512 agcount=4, agsize=65536 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0 spinodes=0
data = bsize=4096 blocks=262144, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
data blocks changed from 262144 to 524288
----------------------------------------------------------------------------------------------
Volume Pool Volume size FS FS size Free Type Mount point
----------------------------------------------------------------------------------------------
/dev/lvm_pool/lvol001 lvm_pool 2.00 GB xfs 1014.00 MB 1013.86 MB linear /media
/dev/vda1 20.00 GB xfs 19.99 GB 18.29 GB part /
----------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------
Volume Pool Volume size FS FS size Free Type Mount point
--------------------------------------------------------------------------------------------
/dev/lvm_pool/lvol001 lvm_pool 2.00 GB xfs 1.99 GB 1013.84 MB linear /media
/dev/vda1 20.00 GB xfs 19.99 GB 18.29 GB part /
--------------------------------------------------------------------------------------------
---
Verified with ssm version 0.4-9.el7
---
[root@host-8-247-225 tests1minutetip_bringup]# ssm create -s 1G --fs xfs /dev/loop0 && lvs && ssm list fs && lvextend -L +1G lvm_pool/lvol001 && lvs && mount /dev/lvm_pool/lvol001 /media && xfs_growfs /media && ssm list fs && sleep 30s && ssm list fs
Physical volume "/dev/loop0" successfully created.
Volume group "lvm_pool" successfully created
Logical volume "lvol001" created.
meta-data=/dev/lvm_pool/lvol001 isize=512 agcount=4, agsize=65536 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0, sparse=0
data = bsize=4096 blocks=262144, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
lvol001 lvm_pool -wi-a----- 1.00g
----------------------------------------------------------------------------------------------
Volume Pool Volume size FS FS size Free Type Mount point
----------------------------------------------------------------------------------------------
/dev/lvm_pool/lvol001 lvm_pool 1.00 GB xfs 1014.00 MB 1013.86 MB linear
/dev/vda1 20.00 GB xfs 19.99 GB 14.28 GB /
----------------------------------------------------------------------------------------------
Size of logical volume lvm_pool/lvol001 changed from 1.00 GiB (256 extents) to 2.00 GiB (512 extents).
Logical volume lvm_pool/lvol001 successfully resized.
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
lvol001 lvm_pool -wi-a----- 2.00g
meta-data=/dev/mapper/lvm_pool-lvol001 isize=512 agcount=4, agsize=65536 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0 spinodes=0
data = bsize=4096 blocks=262144, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
data blocks changed from 262144 to 524288
------------------------------------------------------------------------------------------
Volume Pool Volume size FS FS size Free Type Mount point
------------------------------------------------------------------------------------------
/dev/lvm_pool/lvol001 lvm_pool 2.00 GB xfs 1.99 GB 1.96 GB linear /media
/dev/vda1 20.00 GB xfs 19.99 GB 14.28 GB /
------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------
Volume Pool Volume size FS FS size Free Type Mount point
------------------------------------------------------------------------------------------
/dev/lvm_pool/lvol001 lvm_pool 2.00 GB xfs 1.99 GB 1.96 GB linear /media
/dev/vda1 20.00 GB xfs 19.99 GB 14.28 GB /
------------------------------------------------------------------------------------------
---
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:2133 |