Bug 1524048

Summary: gluster volume set is very slow
Product: [Community] GlusterFS Reporter: nh2 <nh2-redhatbugzilla>
Component: glusterdAssignee: Sanju <srakonde>
Status: CLOSED NOTABUG QA Contact:
Severity: low Docs Contact:
Priority: low    
Version: mainlineCC: bugs, pasik, srakonde
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-09-19 10:56:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description nh2 2017-12-09 16:35:39 UTC 
Description of problem:

Running a `gluster volume set` command takes almost 2.5 seconds for my 3-machine cluster, no matter if these are physical machines in a LAN or VMs or containers on the same host (so there is negligible network latency).

I have a script that needs to set 22 gluster options. As a result, this takes 1 minute. It slows down my automatic deployment, so I'd like it to to be faster.

With some investigation, I found that it's mainly fsync()s that make it this slow.

Example:

    [root@node-3:~]# time gluster vol set myvol rollover-time 1
    volume set: success
    
    real    0m2.120s

Now we set nobarrier on the filesystem, which makes fsyncs no-ops:

    [all-hosts] # mount -o remount,nobarrier /

    [root@node-3:~]# time gluster vol set myvol rollover-time 1
    volume set: success
    
    real    0m0.242s

That's 10x faster.

`strace` confirms that fsync is at play here: In normal gluster idle operation, glusterfsd shows little to no fsync calls. When I execute the `volume set`, I get 3 seconds in `strace -fp "$(pidof glusterfsd)" -wc`:

    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
      1.46    3.028789       84133        36           fsync

How reproducible: Always

Steps to Reproduce:
1. setup any gluster cluster
2. run `time gluster volume set` with some option
3. compare run time to when the file system that carries gluster stuf is mounted nobarrier

Additional info: Using gluster 3.12.3 on Linux
Comment 1 nh2 2017-12-09 16:38:22 UTC 
Here's some more info, showing exactly the fsync syscalls that are happening:

With barrier (normal):

[root@host:~]# mount -o remount,barrier /
[root@host:~]# time strace -fp "$(pidof glusterfsd)" -e fsync -tttT
[pid 27543] 1512837193.344489 fsync(33) = 0 <0.098734>
[pid 27543] 1512837193.443479 fsync(34) = 0 <0.060922>
[pid 27543] 1512837193.507358 fsync(33) = 0 <0.061724>
[pid 27543] 1512837193.569290 fsync(37) = 0 <0.083958>
[pid 27543] 1512837193.653654 fsync(34) = 0 <0.100039>
[pid 27543] 1512837193.767362 fsync(33) = 0 <0.099843>
[pid 27543] 1512837193.867350 fsync(35) = 0 <0.059362>
[pid 27543] 1512837193.926851 fsync(33) = 0 <0.037742>
[pid 27543] 1512837193.964734 fsync(32) = 0 <0.077420>
[pid 27543] 1512837194.042367 fsync(33) = 0 <0.054849>
[pid 27543] 1512837194.097981 fsync(32) = 0 <0.100883>
[pid 27543] 1512837194.199456 fsync(33) = 0 <0.059231>
[pid 27675] 1512837194.285334 fsync(32 <unfinished ...>
 <unfinished ...>
[pid 27675] 1512837194.373005 <... fsync resumed> ) = 0 <0.087665>
[pid 27762] 1512837194.373027 <... fsync resumed> ) = 0 <0.082764>
[pid 27675] 1512837194.373138 fsync(33 <unfinished ...>
[pid 27762] 1512837194.373155 fsync(42) = 0 <0.049708>
[pid 27675] 1512837194.422877 <... fsync resumed> ) = 0 <0.049734>
[pid 27675] 1512837194.425988 fsync(32 <unfinished ...>
[pid 27762] 1512837194.428923 fsync(41 <unfinished ...>
[pid 27675] 1512837194.525484 <... fsync resumed> ) = 0 <0.099484>
[pid 27762] 1512837194.525496 <... fsync resumed> ) = 0 <0.096568>
[pid 27675] 1512837194.525595 fsync(35 <unfinished ...>
[pid 27762] 1512837194.525624 fsync(44 <unfinished ...>
[pid 27675] 1512837194.584304 <... fsync resumed> ) = 0 <0.058690>
[pid 27762] 1512837194.584319 <... fsync resumed> ) = 0 <0.058690>
[pid 27675] 1512837194.584436 fsync(33 <unfinished ...>
[pid 27762] 1512837194.584509 fsync(42 <unfinished ...>
[pid 27675] 1512837194.669038 <... fsync resumed> ) = 0 <0.084582>
[pid 27762] 1512837194.669051 <... fsync resumed> ) = 0 <0.084538>
[pid 27675] 1512837194.669289 fsync(32 <unfinished ...>
[pid 27762] 1512837194.669442 fsync(41 <unfinished ...>
[pid 27675] 1512837194.712706 <... fsync resumed> ) = 0 <0.043400>
[pid 27675] 1512837194.712786 fsync(34 <unfinished ...>
[pid 27762] 1512837194.755490 <... fsync resumed> ) = 0 <0.086025>
[pid 27762] 1512837194.755924 fsync(43 <unfinished ...>
[pid 27675] 1512837194.808851 <... fsync resumed> ) = 0 <0.096050>
[pid 27675] 1512837194.809115 fsync(32 <unfinished ...>
[pid 27762] 1512837194.860723 <... fsync resumed> ) = 0 <0.104776>
[pid 27762] 1512837194.861140 fsync(42 <unfinished ...>
[pid 27675] 1512837194.900553 <... fsync resumed> ) = 0 <0.091421>
[pid 27675] 1512837194.900623 fsync(31 <unfinished ...>
[pid 27762] 1512837194.933434 <... fsync resumed> ) = 0 <0.072278>
[pid 27762] 1512837194.933630 fsync(40 <unfinished ...>
[pid 27675] 1512837194.983271 <... fsync resumed> ) = 0 <0.082632>
[pid 27675] 1512837194.983556 fsync(32 <unfinished ...>
[pid 27762] 1512837195.015543 <... fsync resumed> ) = 0 <0.081842>
[pid 27762] 1512837195.015769 fsync(42 <unfinished ...>
[pid 27675] 1512837195.067165 <... fsync resumed> ) = 0 <0.083596>
[pid 27675] 1512837195.086955 fsync(31 <unfinished ...>
[pid 27762] 1512837195.129398 <... fsync resumed> ) = 0 <0.113616>
[pid 27762] 1512837195.130201 fsync(40 <unfinished ...>
[pid 27675] 1512837195.189593 <... fsync resumed> ) = 0 <0.102624>
[pid 27675] 1512837195.189890 fsync(32 <unfinished ...>
[pid 27762] 1512837195.240467 <... fsync resumed> ) = 0 <0.110255>
[pid 27762] 1512837195.240839 fsync(42 <unfinished ...>
[pid 27675] 1512837195.292182 <... fsync resumed> ) = 0 <0.102274>


With nobarrier:

[root@host:~]# mount -o remount,nobarrier /
[root@host:~]# time strace -fp "$(pidof glusterfsd)" -e fsync -tttT
[pid 27543] 1512837034.233502 fsync(33) = 0 <0.000556>
[pid 27543] 1512837034.234154 fsync(34) = 0 <0.000271>
[pid 27543] 1512837034.237067 fsync(33) = 0 <0.005847>
[pid 27543] 1512837034.243019 fsync(36) = 0 <0.000278>
[pid 27543] 1512837034.243329 fsync(34) = 0 <0.008456>
[pid 27543] 1512837034.251897 fsync(33) = 0 <0.000275>
[pid 27543] 1512837034.252211 fsync(35) = 0 <0.008275>
[pid 27543] 1512837034.260567 fsync(33) = 0 <0.000305>
[pid 27543] 1512837034.260925 fsync(32) = 0 <0.008355>
[pid 27543] 1512837034.269354 fsync(33) = 0 <0.000272>
[pid 27543] 1512837034.270136 fsync(32) = 0 <0.008022>
[pid 27543] 1512837034.278313 fsync(33) = 0 <0.000325>
[pid 27675] 1512837034.297006 fsync(32) = 0 <0.000559>
[pid 27675] 1512837034.297675 fsync(33) = 0 <0.000264>
[pid 27675] 1512837034.301958 fsync(32 <unfinished ...>
 <unfinished ...>
[pid 27675] 1512837034.322711 <... fsync resumed> ) = 0 <0.020749>
[pid 27675] 1512837034.322951 fsync(35 <unfinished ...>
[pid 27762] 1512837034.322960 <... fsync resumed> ) = 0 <0.015717>
[pid 27762] 1512837034.323212 fsync(42 <unfinished ...>
[pid 27675] 1512837034.323232 <... fsync resumed> ) = 0 <0.000276>
[pid 27675] 1512837034.323254 fsync(33 <unfinished ...>
[pid 27762] 1512837034.323459 <... fsync resumed> ) = 0 <0.000242>
[pid 27762] 1512837034.326539 fsync(41 <unfinished ...>
[pid 27675] 1512837034.332323 <... fsync resumed> ) = 0 <0.009061>
[pid 27762] 1512837034.332331 <... fsync resumed> ) = 0 <0.005789>
[pid 27675] 1512837034.332428 fsync(32 <unfinished ...>
[pid 27762] 1512837034.332440 fsync(44 <unfinished ...>
[pid 27675] 1512837034.332785 <... fsync resumed> ) = 0 <0.000349>
[pid 27675] 1512837034.332809 fsync(34 <unfinished ...>
[pid 27762] 1512837034.332817 <... fsync resumed> ) = 0 <0.000374>
[pid 27762] 1512837034.332840 fsync(42 <unfinished ...>
[pid 27675] 1512837034.341147 <... fsync resumed> ) = 0 <0.008331>
[pid 27762] 1512837034.341159 <... fsync resumed> ) = 0 <0.008316>
[pid 27675] 1512837034.341198 fsync(32 <unfinished ...>
[pid 27762] 1512837034.341847 fsync(41 <unfinished ...>
[pid 27675] 1512837034.341892 <... fsync resumed> ) = 0 <0.000690>
[pid 27675] 1512837034.341914 fsync(31 <unfinished ...>
[pid 27762] 1512837034.342140 <... fsync resumed> ) = 0 <0.000289>
[pid 27762] 1512837034.342180 fsync(43 <unfinished ...>
[pid 27675] 1512837034.350519 <... fsync resumed> ) = 0 <0.008599>
[pid 27762] 1512837034.350530 <... fsync resumed> ) = 0 <0.008348>
[pid 27675] 1512837034.350572 fsync(32 <unfinished ...>
[pid 27762] 1512837034.350599 fsync(41 <unfinished ...>
[pid 27675] 1512837034.350878 <... fsync resumed> ) = 0 <0.000302>
[pid 27762] 1512837034.350884 <... fsync resumed> ) = 0 <0.000284>
[pid 27762] 1512837034.350926 fsync(40 <unfinished ...>
[pid 27675] 1512837034.351361 fsync(31) = 0 <0.008223>
[pid 27762] 1512837034.359596 <... fsync resumed> ) = 0 <0.008668>
[pid 27675] 1512837034.359649 fsync(32 <unfinished ...>
[pid 27762] 1512837034.359672 fsync(41 <unfinished ...>
[pid 27675] 1512837034.359956 <... fsync resumed> ) = 0 <0.000300>
[pid 27762] 1512837034.359980 <... fsync resumed> ) = 0 <0.000307>
) = 0 <0.007432>
[pid 27762] 1512837034.368545 fsync(41) = 0 <0.000251>


You can see that without nobarrier, the fsync delays add up to the latency I observe.

Is it really necessary that gluster fsync()s e.g. FD 33 8 times in a row to execute a `volume set`?

Would it be possible to execut an fsync only at the end of the operation, to make `gluster volume set` 10x faster?
Comment 2 nh2 2017-12-09 16:49:12 UTC 
Using gdb, I have captured the stack traces to the 12 fsyncs that one glusterfsd process makes during `gluster volume set`.

Abbreviated and deduplicated, the calls to fsync() are from:

#1  0x00007f32c87f3cab in gf_store_rename_tmppath
#2  0x00007f32bcdedbe4 in glusterd_store_perform_snapd_store

#1  0x00007f32c87f3b6c in gf_store_sync_direntry
#2  0x00007f32c87f3de8 in gf_store_rename_tmppath

#1  0x00007f32c87f3cab in gf_store_rename_tmppath
#2  0x00007f32bcdf0d5f in glusterd_store_brickinfos_atomic_update

#1  0x00007f32c87f3cab in gf_store_rename_tmppath
#2  0x00007f32bcdf0dca in glusterd_store_volinfo_atomic_update

#1  0x00007f32c87f3cab in gf_store_rename_tmppath
#2  0x00007f32bcdf0970 in glusterd_store_perform_node_state_store


The full stack traces of all 12 fsyncs in order are:

#0  0x00007f32c7a21e40 in fsync () from /nix/store/h1a1ncbkkhapzm0509plqjlfrgxw22f3-glibc-2.25-49/lib/libpthread.so.0
#1  0x00007f32c87f3cab in gf_store_rename_tmppath () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/libglusterfs.so.0
#2  0x00007f32bcdedbe4 in glusterd_store_perform_snapd_store () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#3  0x00007f32bcdedf48 in glusterd_store_snapd_info () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#4  0x00007f32bcdee928 in glusterd_volume_write_snap_details () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#5  0x00007f32bcdef2a3 in glusterd_volume_exclude_options_write () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#6  0x00007f32bcdef66c in glusterd_store_volinfo_write () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#7  0x00007f32bcdf0a81 in glusterd_store_perform_volume_store () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#8  0x00007f32bcdf1232 in glusterd_store_volinfo () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#9  0x00007f32bcdb40bd in glusterd_op_commit_perform () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#10 0x00007f32bcdbb6dc in glusterd_op_ac_commit_op () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#11 0x00007f32bcdb891e in glusterd_op_sm () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#12 0x00007f32bcd952b4 in __glusterd_handle_commit_op () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#13 0x00007f32bcd9c63d in glusterd_big_locked_handler () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#14 0x00007f32c87f8fe0 in synctask_wrap () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/libglusterfs.so.0
#15 0x00007f32c727bc60 in ?? () from /nix/store/h1a1ncbkkhapzm0509plqjlfrgxw22f3-glibc-2.25-49/lib/libc.so.6
#16 0x0000000000000000 in ?? ()


#0  0x00007f32c7a21e40 in fsync () from /nix/store/h1a1ncbkkhapzm0509plqjlfrgxw22f3-glibc-2.25-49/lib/libpthread.so.0
#1  0x00007f32c87f3b6c in gf_store_sync_direntry () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/libglusterfs.so.0
#2  0x00007f32c87f3de8 in gf_store_rename_tmppath () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/libglusterfs.so.0
#3  0x00007f32bcdedbe4 in glusterd_store_perform_snapd_store () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#4  0x00007f32bcdedf48 in glusterd_store_snapd_info () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#5  0x00007f32bcdee928 in glusterd_volume_write_snap_details () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#6  0x00007f32bcdef2a3 in glusterd_volume_exclude_options_write () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#7  0x00007f32bcdef66c in glusterd_store_volinfo_write () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#8  0x00007f32bcdf0a81 in glusterd_store_perform_volume_store () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#9  0x00007f32bcdf1232 in glusterd_store_volinfo () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#10 0x00007f32bcdb40bd in glusterd_op_commit_perform () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#11 0x00007f32bcdbb6dc in glusterd_op_ac_commit_op () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#12 0x00007f32bcdb891e in glusterd_op_sm () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#13 0x00007f32bcd952b4 in __glusterd_handle_commit_op () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#14 0x00007f32bcd9c63d in glusterd_big_locked_handler () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#15 0x00007f32c87f8fe0 in synctask_wrap () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/libglusterfs.so.0
#16 0x00007f32c727bc60 in ?? () from /nix/store/h1a1ncbkkhapzm0509plqjlfrgxw22f3-glibc-2.25-49/lib/libc.so.6
#17 0x0000000000000000 in ?? ()


#0  0x00007f32c7a21e40 in fsync () from /nix/store/h1a1ncbkkhapzm0509plqjlfrgxw22f3-glibc-2.25-49/lib/libpthread.so.0
#1  0x00007f32c87f3cab in gf_store_rename_tmppath () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/libglusterfs.so.0
#2  0x00007f32bcdf0d5f in glusterd_store_brickinfos_atomic_update () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#3  0x00007f32bcdf0e81 in glusterd_store_volume_atomic_update () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#4  0x00007f32bcdf1244 in glusterd_store_volinfo () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#5  0x00007f32bcdb40bd in glusterd_op_commit_perform () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#6  0x00007f32bcdbb6dc in glusterd_op_ac_commit_op () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#7  0x00007f32bcdb891e in glusterd_op_sm () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#8  0x00007f32bcd952b4 in __glusterd_handle_commit_op () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#9  0x00007f32bcd9c63d in glusterd_big_locked_handler () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#10 0x00007f32c87f8fe0 in synctask_wrap () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/libglusterfs.so.0
#11 0x00007f32c727bc60 in ?? () from /nix/store/h1a1ncbkkhapzm0509plqjlfrgxw22f3-glibc-2.25-49/lib/libc.so.6
#12 0x0000000000000000 in ?? ()


#0  0x00007f32c7a21e40 in fsync () from /nix/store/h1a1ncbkkhapzm0509plqjlfrgxw22f3-glibc-2.25-49/lib/libpthread.so.0
#1  0x00007f32c87f3b6c in gf_store_sync_direntry () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/libglusterfs.so.0
#2  0x00007f32c87f3de8 in gf_store_rename_tmppath () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/libglusterfs.so.0
#3  0x00007f32bcdf0d5f in glusterd_store_brickinfos_atomic_update () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#4  0x00007f32bcdf0e81 in glusterd_store_volume_atomic_update () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#5  0x00007f32bcdf1244 in glusterd_store_volinfo () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#6  0x00007f32bcdb40bd in glusterd_op_commit_perform () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#7  0x00007f32bcdbb6dc in glusterd_op_ac_commit_op () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#8  0x00007f32bcdb891e in glusterd_op_sm () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#9  0x00007f32bcd952b4 in __glusterd_handle_commit_op () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#10 0x00007f32bcd9c63d in glusterd_big_locked_handler () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#11 0x00007f32c87f8fe0 in synctask_wrap () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/libglusterfs.so.0
#12 0x00007f32c727bc60 in ?? () from /nix/store/h1a1ncbkkhapzm0509plqjlfrgxw22f3-glibc-2.25-49/lib/libc.so.6
#13 0x0000000000000000 in ?? ()


#0  0x00007f32c7a21e40 in fsync () from /nix/store/h1a1ncbkkhapzm0509plqjlfrgxw22f3-glibc-2.25-49/lib/libpthread.so.0
#1  0x00007f32c87f3cab in gf_store_rename_tmppath () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/libglusterfs.so.0
#2  0x00007f32bcdf0d5f in glusterd_store_brickinfos_atomic_update () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#3  0x00007f32bcdf0e81 in glusterd_store_volume_atomic_update () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#4  0x00007f32bcdf1244 in glusterd_store_volinfo () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#5  0x00007f32bcdb40bd in glusterd_op_commit_perform () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#6  0x00007f32bcdbb6dc in glusterd_op_ac_commit_op () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#7  0x00007f32bcdb891e in glusterd_op_sm () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#8  0x00007f32bcd952b4 in __glusterd_handle_commit_op () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#9  0x00007f32bcd9c63d in glusterd_big_locked_handler () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#10 0x00007f32c87f8fe0 in synctask_wrap () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/libglusterfs.so.0
#11 0x00007f32c727bc60 in ?? () from /nix/store/h1a1ncbkkhapzm0509plqjlfrgxw22f3-glibc-2.25-49/lib/libc.so.6
#12 0x0000000000000000 in ?? ()


#0  0x00007f32c7a21e40 in fsync () from /nix/store/h1a1ncbkkhapzm0509plqjlfrgxw22f3-glibc-2.25-49/lib/libpthread.so.0
#1  0x00007f32c87f3b6c in gf_store_sync_direntry () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/libglusterfs.so.0
#2  0x00007f32c87f3de8 in gf_store_rename_tmppath () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/libglusterfs.so.0
#3  0x00007f32bcdf0d5f in glusterd_store_brickinfos_atomic_update () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#4  0x00007f32bcdf0e81 in glusterd_store_volume_atomic_update () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#5  0x00007f32bcdf1244 in glusterd_store_volinfo () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#6  0x00007f32bcdb40bd in glusterd_op_commit_perform () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#7  0x00007f32bcdbb6dc in glusterd_op_ac_commit_op () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#8  0x00007f32bcdb891e in glusterd_op_sm () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#9  0x00007f32bcd952b4 in __glusterd_handle_commit_op () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#10 0x00007f32bcd9c63d in glusterd_big_locked_handler () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#11 0x00007f32c87f8fe0 in synctask_wrap () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/libglusterfs.so.0
#12 0x00007f32c727bc60 in ?? () from /nix/store/h1a1ncbkkhapzm0509plqjlfrgxw22f3-glibc-2.25-49/lib/libc.so.6
#13 0x0000000000000000 in ?? ()


#0  0x00007f32c7a21e40 in fsync () from /nix/store/h1a1ncbkkhapzm0509plqjlfrgxw22f3-glibc-2.25-49/lib/libpthread.so.0
#1  0x00007f32c87f3cab in gf_store_rename_tmppath () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/libglusterfs.so.0
#2  0x00007f32bcdf0d5f in glusterd_store_brickinfos_atomic_update () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#3  0x00007f32bcdf0e81 in glusterd_store_volume_atomic_update () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#4  0x00007f32bcdf1244 in glusterd_store_volinfo () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#5  0x00007f32bcdb40bd in glusterd_op_commit_perform () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#6  0x00007f32bcdbb6dc in glusterd_op_ac_commit_op () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#7  0x00007f32bcdb891e in glusterd_op_sm () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#8  0x00007f32bcd952b4 in __glusterd_handle_commit_op () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#9  0x00007f32bcd9c63d in glusterd_big_locked_handler () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#10 0x00007f32c87f8fe0 in synctask_wrap () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/libglusterfs.so.0
#11 0x00007f32c727bc60 in ?? () from /nix/store/h1a1ncbkkhapzm0509plqjlfrgxw22f3-glibc-2.25-49/lib/libc.so.6
#12 0x0000000000000000 in ?? ()


#0  0x00007f32c7a21e40 in fsync () from /nix/store/h1a1ncbkkhapzm0509plqjlfrgxw22f3-glibc-2.25-49/lib/libpthread.so.0
#1  0x00007f32c87f3b6c in gf_store_sync_direntry () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/libglusterfs.so.0
#2  0x00007f32c87f3de8 in gf_store_rename_tmppath () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/libglusterfs.so.0
#3  0x00007f32bcdf0d5f in glusterd_store_brickinfos_atomic_update () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#4  0x00007f32bcdf0e81 in glusterd_store_volume_atomic_update () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#5  0x00007f32bcdf1244 in glusterd_store_volinfo () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#6  0x00007f32bcdb40bd in glusterd_op_commit_perform () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#7  0x00007f32bcdbb6dc in glusterd_op_ac_commit_op () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#8  0x00007f32bcdb891e in glusterd_op_sm () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#9  0x00007f32bcd952b4 in __glusterd_handle_commit_op () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#10 0x00007f32bcd9c63d in glusterd_big_locked_handler () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#11 0x00007f32c87f8fe0 in synctask_wrap () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/libglusterfs.so.0
#12 0x00007f32c727bc60 in ?? () from /nix/store/h1a1ncbkkhapzm0509plqjlfrgxw22f3-glibc-2.25-49/lib/libc.so.6
#13 0x0000000000000000 in ?? ()


#0  0x00007f32c7a21e40 in fsync () from /nix/store/h1a1ncbkkhapzm0509plqjlfrgxw22f3-glibc-2.25-49/lib/libpthread.so.0
#1  0x00007f32c87f3cab in gf_store_rename_tmppath () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/libglusterfs.so.0
#2  0x00007f32bcdf0dca in glusterd_store_volinfo_atomic_update () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#3  0x00007f32bcdf1244 in glusterd_store_volinfo () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#4  0x00007f32bcdb40bd in glusterd_op_commit_perform () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#5  0x00007f32bcdbb6dc in glusterd_op_ac_commit_op () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#6  0x00007f32bcdb891e in glusterd_op_sm () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#7  0x00007f32bcd952b4 in __glusterd_handle_commit_op () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#8  0x00007f32bcd9c63d in glusterd_big_locked_handler () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#9  0x00007f32c87f8fe0 in synctask_wrap () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/libglusterfs.so.0
#10 0x00007f32c727bc60 in ?? () from /nix/store/h1a1ncbkkhapzm0509plqjlfrgxw22f3-glibc-2.25-49/lib/libc.so.6
#11 0x0000000000000000 in ?? ()


#0  0x00007f32c7a21e40 in fsync () from /nix/store/h1a1ncbkkhapzm0509plqjlfrgxw22f3-glibc-2.25-49/lib/libpthread.so.0
#1  0x00007f32c87f3b6c in gf_store_sync_direntry () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/libglusterfs.so.0
#2  0x00007f32c87f3de8 in gf_store_rename_tmppath () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/libglusterfs.so.0
#3  0x00007f32bcdf0dca in glusterd_store_volinfo_atomic_update () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#4  0x00007f32bcdf1244 in glusterd_store_volinfo () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#5  0x00007f32bcdb40bd in glusterd_op_commit_perform () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#6  0x00007f32bcdbb6dc in glusterd_op_ac_commit_op () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#7  0x00007f32bcdb891e in glusterd_op_sm () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#8  0x00007f32bcd952b4 in __glusterd_handle_commit_op () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#9  0x00007f32bcd9c63d in glusterd_big_locked_handler () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#10 0x00007f32c87f8fe0 in synctask_wrap () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/libglusterfs.so.0
#11 0x00007f32c727bc60 in ?? () from /nix/store/h1a1ncbkkhapzm0509plqjlfrgxw22f3-glibc-2.25-49/lib/libc.so.6
#12 0x0000000000000000 in ?? ()


#0  0x00007f32c7a21e40 in fsync () from /nix/store/h1a1ncbkkhapzm0509plqjlfrgxw22f3-glibc-2.25-49/lib/libpthread.so.0
#1  0x00007f32c87f3cab in gf_store_rename_tmppath () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/libglusterfs.so.0
#2  0x00007f32bcdf0970 in glusterd_store_perform_node_state_store () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#3  0x00007f32bcdf12ae in glusterd_store_volinfo () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#4  0x00007f32bcdb40bd in glusterd_op_commit_perform () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#5  0x00007f32bcdbb6dc in glusterd_op_ac_commit_op () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#6  0x00007f32bcdb891e in glusterd_op_sm () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#7  0x00007f32bcd952b4 in __glusterd_handle_commit_op () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#8  0x00007f32bcd9c63d in glusterd_big_locked_handler () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#9  0x00007f32c87f8fe0 in synctask_wrap () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/libglusterfs.so.0
#10 0x00007f32c727bc60 in ?? () from /nix/store/h1a1ncbkkhapzm0509plqjlfrgxw22f3-glibc-2.25-49/lib/libc.so.6
#11 0x0000000000000000 in ?? ()


#0  0x00007f32c7a21e40 in fsync () from /nix/store/h1a1ncbkkhapzm0509plqjlfrgxw22f3-glibc-2.25-49/lib/libpthread.so.0
#1  0x00007f32c87f3b6c in gf_store_sync_direntry () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/libglusterfs.so.0
#2  0x00007f32c87f3de8 in gf_store_rename_tmppath () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/libglusterfs.so.0
#3  0x00007f32bcdf0970 in glusterd_store_perform_node_state_store () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#4  0x00007f32bcdf12ae in glusterd_store_volinfo () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#5  0x00007f32bcdb40bd in glusterd_op_commit_perform () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#6  0x00007f32bcdbb6dc in glusterd_op_ac_commit_op () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#7  0x00007f32bcdb891e in glusterd_op_sm () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#8  0x00007f32bcd952b4 in __glusterd_handle_commit_op () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#9  0x00007f32bcd9c63d in glusterd_big_locked_handler () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/glusterfs/3.12.3/xlator/mgmt/glusterd.so
#10 0x00007f32c87f8fe0 in synctask_wrap () from /nix/store/y9qg9jan88wnsszmb1badhyfak2znpz7-glusterfs-3.12.3/lib/libglusterfs.so.0
#11 0x00007f32c727bc60 in ?? () from /nix/store/h1a1ncbkkhapzm0509plqjlfrgxw22f3-glibc-2.25-49/lib/libc.so.6
#12 0x0000000000000000 in ?? ()
Comment 3 Yaniv Kaul 2019-04-17 11:57:21 UTC 
I've reported upstream similar issue, happening because of gf_store_save_value() that each open, write, flush and close the file...
Comment 4 Sanju 2019-07-10 09:03:55 UTC 
Hi,

can you please check you still see this latency for "volume set" operaion?

Thanks,
Sanju
Comment 5 Sanju 2019-07-10 09:04:23 UTC 
s/operaion/operation
Comment 6 Sanju 2019-09-19 10:56:51 UTC 
I don't see any latency with "volume set" operation. Closing this as not a bug. Please feel free to reopen the bug if you still face this problem.

Thanks,
Sanju
Comment 7 Red Hat Bugzilla 2023-09-14 04:13:55 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days