Bug 1737141 - read() returns more than file size when using direct I/O
Summary: read() returns more than file size when using direct I/O
Keywords:
Status: CLOSED NEXTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: sharding
Version: 6
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
Assignee: Krutika Dhananjay
QA Contact: bugs@gluster.org
URL:
Whiteboard:
Depends On: 1738419
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-08-02 19:19 UTC by Nir Soffer
Modified: 2019-10-24 09:25 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1738419 (view as bug list)
Environment:
Last Closed: 2019-10-24 09:25:08 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Gluster.org Gerrit 23212 None Merged features/shard: Send correct size when reads are sent beyond file size 2019-10-24 09:25:07 UTC

Description Nir Soffer 2019-08-02 19:19:55 UTC
Description of problem:

When using direct I/O, reading from a file returns more data, padding the file
data with zeroes.

Here is an example.

## On a host mounting gluster using fuse

$ pwd
/rhev/data-center/mnt/glusterSD/voodoo4.tlv.redhat.com:_gv0/de566475-5b67-4987-abf3-3dc98083b44c/dom_md


$ mount | grep glusterfs
voodoo4.tlv.redhat.com:/gv0 on /rhev/data-center/mnt/glusterSD/voodoo4.tlv.redhat.com:_gv0 type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)


$ stat metadata 
  File: metadata
  Size: 501       	Blocks: 1          IO Block: 131072 regular file
Device: 31h/49d	Inode: 13313776956941938127  Links: 1
Access: (0644/-rw-r--r--)  Uid: (   36/    vdsm)   Gid: (   36/     kvm)
Context: system_u:object_r:fusefs_t:s0
Access: 2019-08-01 22:21:49.186381528 +0300
Modify: 2019-08-01 22:21:49.427404135 +0300
Change: 2019-08-01 22:21:49.969739575 +0300
 Birth: -


$ cat metadata 
ALIGNMENT=1048576
BLOCK_SIZE=4096
CLASS=Data
DESCRIPTION=gv0
IOOPTIMEOUTSEC=10
LEASERETRIES=3
LEASETIMESEC=60
LOCKPOLICY=
LOCKRENEWALINTERVALSEC=5
MASTER_VERSION=1
POOL_DESCRIPTION=4k-gluster
POOL_DOMAINS=de566475-5b67-4987-abf3-3dc98083b44c:Active
POOL_SPM_ID=-1
POOL_SPM_LVER=-1
POOL_UUID=44cfb532-3144-48bd-a08c-83065a5a1032
REMOTE_PATH=voodoo4.tlv.redhat.com:/gv0
ROLE=Master
SDUUID=de566475-5b67-4987-abf3-3dc98083b44c
TYPE=GLUSTERFS
VERSION=5
_SHA_CKSUM=3d1cb836f4c93679fc5a4e7218425afe473e3cfa


$ dd if=metadata bs=4096 count=1 of=/dev/null
0+1 records in
0+1 records out
501 bytes copied, 0.000340298 s, 1.5 MB/s


$ dd if=metadata bs=4096 count=1 of=/dev/null iflag=direct
1+0 records in
1+0 records out
4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00398529 s, 1.0 MB/s

Checking the copied data, the actual content of the file is padded
with zeros to 4096 bytes.


## On the one of the gluster nodes

$ pwd
/export/vdo0/brick/de566475-5b67-4987-abf3-3dc98083b44c/dom_md


$ stat metadata 
  File: metadata
  Size: 501       	Blocks: 16         IO Block: 4096   regular file
Device: fd02h/64770d	Inode: 149         Links: 2
Access: (0644/-rw-r--r--)  Uid: (   36/ UNKNOWN)   Gid: (   36/     kvm)
Context: system_u:object_r:usr_t:s0
Access: 2019-08-01 22:21:50.380425478 +0300
Modify: 2019-08-01 22:21:49.427397589 +0300
Change: 2019-08-01 22:21:50.374425302 +0300
 Birth: -


$ dd if=metadata bs=4096 count=1 of=/dev/null
0+1 records in
0+1 records out
501 bytes copied, 0.000991636 s, 505 kB/s


$ dd if=metadata bs=4096 count=1 of=/dev/null iflag=direct
0+1 records in
0+1 records out
501 bytes copied, 0.0011381 s, 440 kB/s

This proves that the issue is in gluster.


# gluster volume info gv0
 
Volume Name: gv0
Type: Replicate
Volume ID: cbc5a2ad-7246-42fc-a78f-70175fb7bf22
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (2 + 1) = 3
Transport-type: tcp
Bricks:
Brick1: voodoo4.tlv.redhat.com:/export/vdo0/brick
Brick2: voodoo5.tlv.redhat.com:/export/vdo0/brick
Brick3: voodoo8.tlv.redhat.com:/export/vdo0/brick (arbiter)
Options Reconfigured:
storage.owner-gid: 36
storage.owner-uid: 36
server.event-threads: 4
client.event-threads: 4
cluster.choose-local: off
user.cifs: off
features.shard: on
cluster.shd-wait-qlength: 10000
cluster.shd-max-threads: 8
cluster.locking-scheme: granular
cluster.data-self-heal-algorithm: full
cluster.server-quorum-type: server
cluster.quorum-type: auto
cluster.eager-lock: enable
network.remote-dio: disable
performance.low-prio-threads: 32
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: on


$ xfs_info /export/vdo0
meta-data=/dev/mapper/vdo0       isize=512    agcount=4, agsize=6553600 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=0
data     =                       bsize=4096   blocks=26214400, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=12800, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0


Version-Release number of selected component (if applicable):

Server:

$ rpm -qa | grep glusterfs
glusterfs-libs-6.4-1.fc29.x86_64
glusterfs-api-6.4-1.fc29.x86_64
glusterfs-client-xlators-6.4-1.fc29.x86_64
glusterfs-fuse-6.4-1.fc29.x86_64
glusterfs-6.4-1.fc29.x86_64
glusterfs-cli-6.4-1.fc29.x86_64
glusterfs-server-6.4-1.fc29.x86_64

Client:

$ rpm -qa | grep glusterfs
glusterfs-client-xlators-6.4-1.fc29.x86_64
glusterfs-6.4-1.fc29.x86_64
glusterfs-rdma-6.4-1.fc29.x86_64
glusterfs-cli-6.4-1.fc29.x86_64
glusterfs-libs-6.4-1.fc29.x86_64
glusterfs-fuse-6.4-1.fc29.x86_64
glusterfs-api-6.4-1.fc29.x86_64


How reproducible:
Always.

Steps to Reproduce:
1. Provision gluster volume over vdo (did not check without vdo)
2. Create a file of 501 bytes
3. Read the file using direct I/O

Actual results:
read() returns 4096 bytes, padding the file data with zeroes

Expected results:
read() returns actual file data (501 bytes)

Comment 1 Nir Soffer 2019-08-02 19:21:20 UTC
David, do you think this can affect sanlock?

Comment 2 Nir Soffer 2019-08-02 19:25:02 UTC
Kevin, do you think this can affect qemu/qemu-img?

Comment 3 Amar Tumballi 2019-08-05 05:33:57 UTC
@Nir, thanks for the report. We will look into this.

Comment 4 Kevin Wolf 2019-08-05 09:16:16 UTC
(In reply to Nir Soffer from comment #2)
> Kevin, do you think this can affect qemu/qemu-img?

This is not a problem for QEMU as long as the file size is correct. If gluster didn't do the zero padding, QEMU would do it internally.

In fact, fixing this in gluster may break the case of unaligned image sizes with QEMU because the image size is rounded up to sector (512 byte) granularity and the gluster driver turns short reads into errors. This would actually affect non-O_DIRECT, too, which already seems to behave this way, so can you just give this a quick test?

Comment 5 David Teigland 2019-08-05 15:08:32 UTC
(In reply to Nir Soffer from comment #1)
> David, do you think this can affect sanlock?

I don't think so.  sanlock doesn't use any space that it didn't first write to initialize.

Comment 6 Worker Ant 2019-08-12 15:51:39 UTC
REVIEW: https://review.gluster.org/23212 (features/shard: Send correct size when reads are sent beyond file size) posted (#1) for review on release-6 by Krutika Dhananjay

Comment 7 Worker Ant 2019-10-24 09:25:08 UTC
REVIEW: https://review.gluster.org/23212 (features/shard: Send correct size when reads are sent beyond file size) merged (#3) on release-6 by hari gowtham


Note You need to log in before you can comment on or make changes to this bug.