Bug 1594406

Summary: VDO volume with 4096 byte block size shows an increasing "bios acknowledged partial write" statistic
Product: Red Hat Enterprise Linux 7 Reporter: Bryan Gurney <bgurney>
Component: kmod-kvdoAssignee: sclafani
Status: CLOSED ERRATA QA Contact: Jakub Krysl <jkrysl>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.5CC: awalsh, jkrysl, limershe, sweettea
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: 6.1.2.25 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1648496 (view as bug list) Environment:
Last Closed: 2019-08-06 13:08:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1648496    
Attachments:
Description Flags
Reproduction demonstration with filtered vdostats output
none
Reproduction demonstration with full vdostats output none

Description Bryan Gurney 2018-06-22 21:04:51 UTC 
Description of problem:
It's possible for a VDO volume with a block size of 4096 bytes (i.e.: "--emulate512=disabled", the default setting) to exhibit an increasing "bios acknowledged partial read" statistic, despite the inability for a smaller-than-4096-byte read on the VDO volume (i.e.: any read attempts at smaller than 4096 bytes would fail with -EINVAL, or "invalid argument").

Version-Release number of selected component (if applicable):
kernel-3.10.0-862.3.3.el7.x86_64
kmod-kvdo-6.1.0.168-16.el7_5.x86_64
vdo-6.1.0.168-18.x86_64


How reproducible:
100% so far.

Steps to Reproduce:
1. Use the following fio job to write a test file that contains a duplicate pattern (I used "z" characters):

[globals]

bs=4096
rw=write
name=write_8k_z
numjobs=1
size=8k
norandommap
randrepeat=1
group_reporting
unlink=0
direct=1
iodepth=128
iodepth_batch_complete=16
iodepth_batch_submit=16
ioengine=libaio
buffer_pattern="z"

[thread1]
filename=8k_z.bin


2. Create a VDO volume on a test device (my example is /dev/sdb):
# vdo create --name=vdo1 --device=/dev/sdb

3. Run "vdostats --verbose" to track the current VDO statistics.
# vdostats vdo1 --verbose | egrep 'bios.*read|bios.*write'

4. Write the test file "8k_z.bin" to the first 8 kilobytes of the VDO volume:
# dd if=8k_z.bin of=/dev/mapper/vdo1 bs=4096 oflag=direct

5. Run "vdostats --verbose" to track the current VDO statistics.
# vdostats vdo1 --verbose | egrep 'bios.*read|bios.*write'

6. Run "blkid -p" to run a low-level superblock probe on the VDO volume.
# blkid -p /dev/mapper/vdo1

7. Run "vdostats --verbose" to track the current VDO statistics.
# vdostats vdo1 --verbose | egrep 'bios.*read|bios.*write'

8. Repeat steps 6, then 7, as many times as desired.

Actual results:
Each time "blkid -p" is executed on the VDO volume, the stats for "bios acknowledged partial read" increases by 2, despite the fact that "Emulate 512 byte" is disabled, and that "bios in partial read" has not increased (and is hopefully zero).

Expected results:
The "bios acknowledged partial read" statistic does not increase.

Additional info:
blkid is reading the areas where there are known superblocks for filesystems, logical volume managers, etc.  During each blkid run, you should see the "bios in read" statistic increase by 266.  In this test, since logical blocks 0 and 1 are being written, these blocks will be populated with data.  I saw that "bios out read" and "bios out completed read" seemed to increase by 2 each time "blkid -p" was executed.
Comment 2 Bryan Gurney 2018-06-22 21:13:36 UTC 
Created attachment 1453855 [details]
Reproduction demonstration with filtered vdostats output
Comment 3 sclafani 2018-06-22 21:43:08 UTC 
(In reply to Bryan Gurney from comment #2)
> Reproduction demonstration with filtered vdostats output

Any chance you could attach unfiltered stats? There could be an important clue in there.
Comment 4 sclafani 2018-06-22 22:53:32 UTC 
I wasn't able to repro this on head with kernel 3.10.0-862.2.3.el7.x86_64 running on an lfarm (sync mode).
Comment 5 Bryan Gurney 2018-06-25 14:12:36 UTC 
Created attachment 1454372 [details]
Reproduction demonstration with full vdostats output
Comment 10 Jakub Krysl 2019-04-29 15:42:26 UTC 
vdo-6.1.2.41-4.el7.x86_64
The partial read statisticks no longer increase, setting to verified.

# vdostats vdo1 --verbose | egrep 'bios.*read|bios.*write'
  bios in read                        : 1305
  bios in write                       : 2
  bios in partial read                : 0
  bios in partial write               : 0
  bios out read                       : 6
  bios out write                      : 1
  bios meta read                      : 5
  bios meta write                     : 73
  bios journal read                   : 0
  bios journal write                  : 8
  bios page cache read                : 1
  bios page cache write               : 0
  bios out completed read             : 6
  bios out completed write            : 1
  bios meta completed read            : 5
  bios meta completed write           : 73
  bios journal completed read         : 0
  bios journal completed write        : 8
  bios page cache completed read      : 1
  bios page cache completed write     : 0
  bios acknowledged read              : 1305
  bios acknowledged write             : 2
  bios acknowledged partial read      : 0
  bios acknowledged partial write     : 0
  bios in progress read               : 0
  bios in progress write              : 0

# blkid -p /dev/mapper/vdo1

# vdostats vdo1 --verbose | egrep 'bios.*read|bios.*write'
  bios in read                        : 1566
  bios in write                       : 2
  bios in partial read                : 0
  bios in partial write               : 0
  bios out read                       : 8
  bios out write                      : 1
  bios meta read                      : 5
  bios meta write                     : 73
  bios journal read                   : 0
  bios journal write                  : 8
  bios page cache read                : 1
  bios page cache write               : 0
  bios out completed read             : 8
  bios out completed write            : 1
  bios meta completed read            : 5
  bios meta completed write           : 73
  bios journal completed read         : 0
  bios journal completed write        : 8
  bios page cache completed read      : 1
  bios page cache completed write     : 0
  bios acknowledged read              : 1566
  bios acknowledged write             : 2
  bios acknowledged partial read      : 0
  bios acknowledged partial write     : 0
  bios in progress read               : 0
  bios in progress write              : 0
Comment 12 errata-xmlrpc 2019-08-06 13:08:04 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2233