Bug 1787993

Tagged component(io-cache) as i don't know below issue related to which component in glusterfs.


Description of problem:
We are running etcd in kubernetes with glusterfs as backend storage.

Version-Release number of selected component (if applicable):
Glusterfs 5.9
Etcd - 3.4.3
kubernetes- 1.9.1


How reproducible:
Mount etcd-pod to glusterfs in kubernetes.
Continuously keep writing key to etcd database using the following command.

while [ 1 ]; do dd if=/dev/urandom bs=1024 count=1024 | ETCDCTL_API=3 etcdctl put key || break; done

Check the logs of etcd using kubectl logs -f <etcd-pod>
you will get to see below warning.

2020-01-03 11:50:06.591364 W | etcdserver: request "header:<ID:2814309323035058006 > put:<key:"key" value_size:1048576 >" with result "size:5" took too long (1.052778581s) to execute
2020-01-03 11:50:07.529154 W | etcdserver: request "header:<ID:2814309323035058007 > put:<key:"key" value_size:1048576 >" with result "size:5" took too long (775.427962ms) to execute

Actual results:
Performance is decreasing when running for long time.


Expected results:
Shouldn't see above warning , writing to glusterfs mount should be fast and smooth.

Additional info:
We are running kafka , logstash and influxdb also in kubernetes with glusterfs as backed storage .

we have tested below scenarios

case 1- we have mounted etcd to hostpath and kafka to glusterfs
we are not seeing above issue(warning) , performance was good.

case2 - etcd with glusterfs and kafka hostpath
seeing above issue , performance is very bad.

Note: we haven't not tunned any glusterfs parameters explicitly 
we are running glusterfs with default configurations( as what glusterfs comes up with ).