Bug 1503079

+++ This bug was initially created as a clone of Bug #1501309 +++

+++ This bug was initially created as a clone of Bug #1497156 +++

Description of problem:
While running fio workload on the vms created in RHHI setup i see that fio job gives the below error in json.out file.

json.out file:
=======================
ostname=dhcp35-71.lab.eng.blr.redhat.com, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.7, flags=1
hostname=dhcp35-168.lab.eng.blr.redhat.com, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.7, flags=1
hostname=dhcp35-141.lab.eng.blr.redhat.com, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.7, flags=1
<dhcp35-168.lab.eng.blr.redhat.com> file:filesetup.c:270, func=fstat, error=Input/output error
<dhcp35-168.lab.eng.blr.redhat.com> file:filesetup.c:270, func=fstat, error=Input/output error
<dhcp35-168.lab.eng.blr.redhat.com> file:filesetup.c:270, func=fstat, error=Input/output error
<dhcp35-168.lab.eng.blr.redhat.com> file:filesetup.c:270, func=fstat, error=Input/output error
<dhcp35-168.lab.eng.blr.redhat.com> file:filesetup.c:270, func=fstat, error=Input/output error
<dhcp35-168.lab.eng.blr.redhat.com> file:filesetup.c:270, func=fstat, error=Input/output error
<dhcp35-168.lab.eng.blr.redhat.com> {
<dhcp35-168.lab.eng.blr.redhat.com>   "fio version" : "fio-2.1.7",
<dhcp35-168.lab.eng.blr.redhat.com>   "jobs" : [
<dhcp35-168.lab.eng.blr.redhat.com>
<dhcp35-168.lab.eng.blr.redhat.com>   ]
<dhcp35-168.lab.eng.blr.redhat.com> }

Run status group 0 (all jobs):
client <10.70.35.168>: exited with error 1
<dhcp35-71.lab.eng.blr.redhat.com> fio: pid=0, err=5/file:filesetup.c:174, func=fsync, error=Input/output error
<dhcp35-71.lab.eng.blr.redhat.com> {
<dhcp35-71.lab.eng.blr.redhat.com>   "fio version" : "fio-2.1.7",
<dhcp35-71.lab.eng.blr.redhat.com>   "jobs" : [
<dhcp35-71.lab.eng.blr.redhat.com>
<dhcp35-71.lab.eng.blr.redhat.com>   ]
<dhcp35-71.lab.eng.blr.redhat.com> }

Run status group 0 (all jobs):
client <10.70.35.71>: exited with error 1
<dhcp35-141.lab.eng.blr.redhat.com> fio: pid=0, err=5/file:filesetup.c:174, func=fsync, error=Input/output error
<dhcp35-141.lab.eng.blr.redhat.com> {
<dhcp35-141.lab.eng.blr.redhat.com>   "fio version" : "fio-2.1.7",
<dhcp35-141.lab.eng.blr.redhat.com>   "jobs" : [
<dhcp35-141.lab.eng.blr.redhat.com>
<dhcp35-141.lab.eng.blr.redhat.com>   ]
<dhcp35-141.lab.eng.blr.redhat.com> }

Run status group 0 (all jobs):
client <10.70.35.141>: exited with error 1
{
  "fio version" : "fio-2.1.7",
  "client_stats" : [
    {
      "jobname" : "workload",
      "groupid" : 0,

Attaching the error which is see on the vm console during fio run.

Version-Release number of selected component (if applicable):
glusterfs-3.8.4-45.el7rhgs.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Create RHHI setup with latest glusterfs & RHV bits
2. Create 3 app vms.
3. start running fio.

Actual results:
Running fio results in fsync, error=input/output error.

Expected results:
fsync should run successfully.

Additional info:

screenshot for the vm console when fio runs is captured .
attaching json.out file 
attaching all the sos reports.

--- Additional comment from Sahina Bose on 2017-10-11 07:25:31 EDT ---

Are using gfapi access for VM disks?

--- Additional comment from RamaKasturi on 2017-10-11 10:52:22 EDT ---

sahina, i was seeing this issue with fuse access.