Bug 1787993 - etcd is giving warning key took too long to execute with glusterfs as backend storage when running in production environment
Summary: etcd is giving warning key took too long to execute with glusterfs as backend...
Keywords:
Status: CLOSED UPSTREAM
Alias: None
Product: GlusterFS
Classification: Community
Component: io-cache
Version: 5
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: ---
Assignee: bugs@gluster.org
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-01-06 04:25 UTC by sharmaakshay890
Modified: 2020-03-12 12:19 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: ---
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-03-12 12:19:15 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:


Attachments (Terms of Use)

Description sharmaakshay890 2020-01-06 04:25:06 UTC
Tagged component(io-cache) as i don't know below issue related to which component in glusterfs.


Description of problem:
We are running etcd in kubernetes with glusterfs as backend storage.

Version-Release number of selected component (if applicable):
Glusterfs 5.9
Etcd - 3.4.3
kubernetes- 1.9.1


How reproducible:
Mount etcd-pod to glusterfs in kubernetes.
Continuously keep writing key to etcd database using the following command.

while [ 1 ]; do dd if=/dev/urandom bs=1024 count=1024 | ETCDCTL_API=3 etcdctl put key || break; done

Check the logs of etcd using kubectl logs -f <etcd-pod>
you will get to see below warning.

2020-01-03 11:50:06.591364 W | etcdserver: request "header:<ID:2814309323035058006 > put:<key:"key" value_size:1048576 >" with result "size:5" took too long (1.052778581s) to execute
2020-01-03 11:50:07.529154 W | etcdserver: request "header:<ID:2814309323035058007 > put:<key:"key" value_size:1048576 >" with result "size:5" took too long (775.427962ms) to execute

Actual results:
Performance is decreasing when running for long time.


Expected results:
Shouldn't see above warning , writing to glusterfs mount should be fast and smooth.

Additional info:
We are running kafka , logstash and influxdb also in kubernetes with glusterfs as backed storage .

we have tested below scenarios

case 1- we have mounted etcd to hostpath and kafka to glusterfs
we are not seeing above issue(warning) , performance was good.

case2 - etcd with glusterfs and kafka hostpath
seeing above issue , performance is very bad.

Note: we haven't not tunned any glusterfs parameters explicitly 
we are running glusterfs with default configurations( as what glusterfs comes up with ).

Comment 1 Worker Ant 2020-03-12 12:19:15 UTC
This bug is moved to https://github.com/gluster/glusterfs/issues/870, and will be tracked there from now on. Visit GitHub issues URL for further details


Note You need to log in before you can comment on or make changes to this bug.