Created attachment 998049 [details] screenshot of ovirt events window showing the timeframe (EST) of the issue Description of problem: Environment - converged ovirt (3.5)+glusterfs (3.6.2) When I create multiple vms from template I see self heal issues from a vol heal <vol> info command, and the vm creation itself fails. Version-Release number of selected component (if applicable): How reproducible: 2 in 3 attempts to build using templates failed. I believe the reason 1 run worked was the filesystem cache circumvented I/O to the gluster brick, reducing load. Steps to Reproduce: 1. create ovirt3.5/glusterfs 3.6.2 environment 2. create a template to use to build vms 3. Clear the cache on the converged nodes (drop_caches) 4. use the "new vm" wizard to create the 3 vms - ensuring the storage allocation policy is clone Actual results: Vm creation for all 3 fail, and a vol heal command issued during the run shows something like [root@gprfc085 ~]# date && gluster vol heal vmdomain info Tue Mar 3 14:41:45 EST 2015 Brick gprfc085.sbu.lab.eng.bos.redhat.com:/glusterfs/brick1/vmdomain/ /2ad90339-5c1b-4b0e-b728-3df651ecd025/images/66838053-2c16-41c7-ae54-47bd11ecf9c4/0dfb8a92-2e57-46a3-8d88-6c29419152fc - Possibly undergoing heal Number of entries: 1 Brick gprfc086.sbu.lab.eng.bos.redhat.com:/glusterfs/brick1/vmdomain/ /2ad90339-5c1b-4b0e-b728-3df651ecd025/images/66838053-2c16-41c7-ae54-47bd11ecf9c4/0dfb8a92-2e57-46a3-8d88-6c29419152fc - Possibly undergoing heal Number of entries: 1 Brick gprfc087.sbu.lab.eng.bos.redhat.com:/glusterfs/brick1/vmdomain/ /2ad90339-5c1b-4b0e-b728-3df651ecd025/images/66838053-2c16-41c7-ae54-47bd11ecf9c4/0dfb8a92-2e57-46a3-8d88-6c29419152fc - Possibly undergoing heal Number of entries: 1 [root@gprfc085 ~]# Expected results: vm creation from clone should work whenever the volume is online - if multiple requests are made they should all work at the expense of extending the runtime Additional info:
Created attachment 998050 [details] gluster logs from gprfc085 (minus the cli.log)
Created attachment 998051 [details] gluster logs from gprfc086 (minus the cli.log)
Created attachment 998052 [details] gluster logs from gprfc087 (minus the cli.log)
Kasturi, is this still an issue glusterfs 3.7.14
(In reply to Sahina Bose from comment #4) > Kasturi, is this still an issue glusterfs 3.7.14 Ping?
Will check on this and get back on monday.
Did the following steps to reproduce the bug and i do not hit the issue any more. 1) Created a vm and created template out of it. 2) Dropped cache on the hosts by running the command 'echo 3 > /proc/sys/vm/drop_caches' 3) Created three vms by selecting the storage allocation policy as 'clone' I see that vms were created successfully and heal info shows zero entries. [root@rhsqa-grafton1 ~]# gluster volume heal vmstore info Brick 10.70.36.79:/gluster_bricks/vmstore/vmstore Status: Connected Number of entries: 0 Brick 10.70.36.80:/gluster_bricks/vmstore/vmstore Status: Connected Number of entries: 0 Brick 10.70.36.81:/gluster_bricks/vmstore/vmstore Status: Connected Number of entries: 0
Not seen in last 2 years!