Bug 1594406 - VDO volume with 4096 byte block size shows an increasing "bios acknowledged partial write" statistic
Summary: VDO volume with 4096 byte block size shows an increasing "bios acknowledged p...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: kmod-kvdo
Version: 7.5
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: sclafani
QA Contact: Jakub Krysl
URL:
Whiteboard:
Depends On:
Blocks: 1648496
TreeView+ depends on / blocked
 
Reported: 2018-06-22 21:04 UTC by Bryan Gurney
Modified: 2019-08-06 13:08 UTC (History)
4 users (show)

Fixed In Version: 6.1.2.25
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1648496 (view as bug list)
Environment:
Last Closed: 2019-08-06 13:08:04 UTC
Target Upstream Version:


Attachments (Terms of Use)
Reproduction demonstration with filtered vdostats output (6.40 KB, text/plain)
2018-06-22 21:13 UTC, Bryan Gurney
no flags Details
Reproduction demonstration with full vdostats output (20.19 KB, text/plain)
2018-06-25 14:12 UTC, Bryan Gurney
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:2233 None None None 2019-08-06 13:08:14 UTC

Description Bryan Gurney 2018-06-22 21:04:51 UTC
Description of problem:
It's possible for a VDO volume with a block size of 4096 bytes (i.e.: "--emulate512=disabled", the default setting) to exhibit an increasing "bios acknowledged partial read" statistic, despite the inability for a smaller-than-4096-byte read on the VDO volume (i.e.: any read attempts at smaller than 4096 bytes would fail with -EINVAL, or "invalid argument").

Version-Release number of selected component (if applicable):
kernel-3.10.0-862.3.3.el7.x86_64
kmod-kvdo-6.1.0.168-16.el7_5.x86_64
vdo-6.1.0.168-18.x86_64


How reproducible:
100% so far.

Steps to Reproduce:
1. Use the following fio job to write a test file that contains a duplicate pattern (I used "z" characters):

[globals]

bs=4096
rw=write
name=write_8k_z
numjobs=1
size=8k
norandommap
randrepeat=1
group_reporting
unlink=0
direct=1
iodepth=128
iodepth_batch_complete=16
iodepth_batch_submit=16
ioengine=libaio
buffer_pattern="z"

[thread1]
filename=8k_z.bin


2. Create a VDO volume on a test device (my example is /dev/sdb):
# vdo create --name=vdo1 --device=/dev/sdb

3. Run "vdostats --verbose" to track the current VDO statistics.
# vdostats vdo1 --verbose | egrep 'bios.*read|bios.*write'

4. Write the test file "8k_z.bin" to the first 8 kilobytes of the VDO volume:
# dd if=8k_z.bin of=/dev/mapper/vdo1 bs=4096 oflag=direct

5. Run "vdostats --verbose" to track the current VDO statistics.
# vdostats vdo1 --verbose | egrep 'bios.*read|bios.*write'

6. Run "blkid -p" to run a low-level superblock probe on the VDO volume.
# blkid -p /dev/mapper/vdo1

7. Run "vdostats --verbose" to track the current VDO statistics.
# vdostats vdo1 --verbose | egrep 'bios.*read|bios.*write'

8. Repeat steps 6, then 7, as many times as desired.

Actual results:
Each time "blkid -p" is executed on the VDO volume, the stats for "bios acknowledged partial read" increases by 2, despite the fact that "Emulate 512 byte" is disabled, and that "bios in partial read" has not increased (and is hopefully zero).

Expected results:
The "bios acknowledged partial read" statistic does not increase.

Additional info:
blkid is reading the areas where there are known superblocks for filesystems, logical volume managers, etc.  During each blkid run, you should see the "bios in read" statistic increase by 266.  In this test, since logical blocks 0 and 1 are being written, these blocks will be populated with data.  I saw that "bios out read" and "bios out completed read" seemed to increase by 2 each time "blkid -p" was executed.

Comment 2 Bryan Gurney 2018-06-22 21:13:36 UTC
Created attachment 1453855 [details]
Reproduction demonstration with filtered vdostats output

Comment 3 sclafani 2018-06-22 21:43:08 UTC
(In reply to Bryan Gurney from comment #2)
> Reproduction demonstration with filtered vdostats output

Any chance you could attach unfiltered stats? There could be an important clue in there.

Comment 4 sclafani 2018-06-22 22:53:32 UTC
I wasn't able to repro this on head with kernel 3.10.0-862.2.3.el7.x86_64 running on an lfarm (sync mode).

Comment 5 Bryan Gurney 2018-06-25 14:12:36 UTC
Created attachment 1454372 [details]
Reproduction demonstration with full vdostats output

Comment 10 Jakub Krysl 2019-04-29 15:42:26 UTC
vdo-6.1.2.41-4.el7.x86_64
The partial read statisticks no longer increase, setting to verified.

# vdostats vdo1 --verbose | egrep 'bios.*read|bios.*write'
  bios in read                        : 1305
  bios in write                       : 2
  bios in partial read                : 0
  bios in partial write               : 0
  bios out read                       : 6
  bios out write                      : 1
  bios meta read                      : 5
  bios meta write                     : 73
  bios journal read                   : 0
  bios journal write                  : 8
  bios page cache read                : 1
  bios page cache write               : 0
  bios out completed read             : 6
  bios out completed write            : 1
  bios meta completed read            : 5
  bios meta completed write           : 73
  bios journal completed read         : 0
  bios journal completed write        : 8
  bios page cache completed read      : 1
  bios page cache completed write     : 0
  bios acknowledged read              : 1305
  bios acknowledged write             : 2
  bios acknowledged partial read      : 0
  bios acknowledged partial write     : 0
  bios in progress read               : 0
  bios in progress write              : 0

# blkid -p /dev/mapper/vdo1

# vdostats vdo1 --verbose | egrep 'bios.*read|bios.*write'
  bios in read                        : 1566
  bios in write                       : 2
  bios in partial read                : 0
  bios in partial write               : 0
  bios out read                       : 8
  bios out write                      : 1
  bios meta read                      : 5
  bios meta write                     : 73
  bios journal read                   : 0
  bios journal write                  : 8
  bios page cache read                : 1
  bios page cache write               : 0
  bios out completed read             : 8
  bios out completed write            : 1
  bios meta completed read            : 5
  bios meta completed write           : 73
  bios journal completed read         : 0
  bios journal completed write        : 8
  bios page cache completed read      : 1
  bios page cache completed write     : 0
  bios acknowledged read              : 1566
  bios acknowledged write             : 2
  bios acknowledged partial read      : 0
  bios acknowledged partial write     : 0
  bios in progress read               : 0
  bios in progress write              : 0

Comment 12 errata-xmlrpc 2019-08-06 13:08:04 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2233


Note You need to log in before you can comment on or make changes to this bug.