Bug 1399105
| Summary: | possible memory leak on client when writing to a file while another client issues a truncate | |||
|---|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Nag Pavan Chilakam <nchilaka> | |
| Component: | disperse | Assignee: | Ashish Pandey <aspandey> | |
| Status: | CLOSED ERRATA | QA Contact: | Nag Pavan Chilakam <nchilaka> | |
| Severity: | urgent | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | rhgs-3.2 | CC: | amukherj, rhinduja, rhs-bugs, storage-qa-internal | |
| Target Milestone: | --- | |||
| Target Release: | RHGS 3.2.0 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | glusterfs-3.8.4-7 | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1400818 (view as bug list) | Environment: | ||
| Last Closed: | 2017-03-23 05:51:50 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1351528, 1400818, 1400833 | |||
|
Description
Nag Pavan Chilakam
2016-11-28 09:59:36 UTC
root@dhcp35-37 ~]# gluster v status Status of volume: disperse Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.37:/rhs/brick2/disperse 49153 0 Y 32685 Brick 10.70.35.116:/rhs/brick2/disperse 49152 0 Y 13804 Brick 10.70.35.239:/rhs/brick2/disperse 49152 0 Y 6677 Brick 10.70.35.135:/rhs/brick2/disperse 49153 0 Y 6698 Brick 10.70.35.8:/rhs/brick2/disperse 49153 0 Y 6617 Brick 10.70.35.196:/rhs/brick2/disperse 49153 0 Y 6700 Brick 10.70.35.37:/rhs/brick3/disperse 49154 0 Y 32704 Brick 10.70.35.116:/rhs/brick3/disperse 49154 0 Y 13823 Brick 10.70.35.239:/rhs/brick3/disperse 49154 0 Y 6696 Brick 10.70.35.135:/rhs/brick3/disperse 49154 0 Y 6717 Brick 10.70.35.8:/rhs/brick3/disperse 49154 0 Y 6636 Brick 10.70.35.196:/rhs/brick3/disperse 49154 0 Y 6719 Self-heal Daemon on localhost N/A N/A Y 32724 Self-heal Daemon on 10.70.35.239 N/A N/A Y 6716 Self-heal Daemon on 10.70.35.8 N/A N/A Y 6656 Self-heal Daemon on 10.70.35.196 N/A N/A Y 6742 Self-heal Daemon on 10.70.35.135 N/A N/A Y 6737 Self-heal Daemon on 10.70.35.116 N/A N/A Y 13843 Task Status of Volume disperse ------------------------------------------------------------------------------ Task : Rebalance ID : e401b7fd-1ef6-4d1f-a818-228cba36a128 Status : completed [root@dhcp35-37 ~]# gluster v info disperse Volume Name: disperse Type: Distributed-Disperse Volume ID: 9620f72c-f8bb-4e05-98e4-d1a2132db8a7 Status: Started Snapshot Count: 0 Number of Bricks: 2 x (4 + 2) = 12 Transport-type: tcp Bricks: Brick1: 10.70.35.37:/rhs/brick2/disperse Brick2: 10.70.35.116:/rhs/brick2/disperse Brick3: 10.70.35.239:/rhs/brick2/disperse Brick4: 10.70.35.135:/rhs/brick2/disperse Brick5: 10.70.35.8:/rhs/brick2/disperse Brick6: 10.70.35.196:/rhs/brick2/disperse Brick7: 10.70.35.37:/rhs/brick3/disperse Brick8: 10.70.35.116:/rhs/brick3/disperse Brick9: 10.70.35.239:/rhs/brick3/disperse Brick10: 10.70.35.135:/rhs/brick3/disperse Brick11: 10.70.35.8:/rhs/brick3/disperse Brick12: 10.70.35.196:/rhs/brick3/disperse Options Reconfigured: transport.address-family: inet performance.readdir-ahead: on nfs.disable: on [root@dhcp35-37 ~]# Starting volume profile on disperse has been successful
[root@dhcp35-37 ~]# gluster v profile disperse info
Brick: 10.70.35.37:/rhs/brick3/disperse
---------------------------------------
Cumulative Stats:
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 901380 0 0
No. of Writes: 70203 1693147 573317
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 0 0 0
No. of Writes: 3144 2175 656
Block Size: 32768b+
No. of Reads: 0
No. of Writes: 262
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 62 FORGET
0.00 0.00 us 0.00 us 0.00 us 1175 RELEASE
0.00 0.00 us 0.00 us 0.00 us 1957 RELEASEDIR
1.50 97.00 us 97.00 us 97.00 us 1 OPENDIR
1.53 99.00 us 99.00 us 99.00 us 1 GETXATTR
7.77 252.00 us 43.00 us 461.00 us 2 READDIR
34.84 150.67 us 76.00 us 600.00 us 15 LOOKUP
54.37 117.57 us 23.00 us 835.00 us 30 INODELK
Duration: 15820 seconds
Data Read: 461506560 bytes
Data Written: 3447035392 bytes
Interval 0 Stats:
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 901380 0 0
No. of Writes: 70203 1693147 573317
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 0 0 0
No. of Writes: 3144 2175 656
Block Size: 32768b+
No. of Reads: 0
No. of Writes: 262
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 62 FORGET
0.00 0.00 us 0.00 us 0.00 us 1175 RELEASE
0.00 0.00 us 0.00 us 0.00 us 1957 RELEASEDIR
1.50 97.00 us 97.00 us 97.00 us 1 OPENDIR
1.53 99.00 us 99.00 us 99.00 us 1 GETXATTR
7.77 252.00 us 43.00 us 461.00 us 2 READDIR
34.84 150.67 us 76.00 us 600.00 us 15 LOOKUP
54.37 117.57 us 23.00 us 835.00 us 30 INODELK
Duration: 15820 seconds
Data Read: 461506560 bytes
Data Written: 3447035392 bytes
Brick: 10.70.35.37:/rhs/brick2/disperse
---------------------------------------
Cumulative Stats:
Block Size: 1b+ 512b+ 1024b+
No. of Reads: 0 2064583 0
No. of Writes: 3 99280 2270930
Block Size: 2048b+ 4096b+ 8192b+
No. of Reads: 0 0 0
No. of Writes: 1699103 6563 5390
Block Size: 16384b+ 32768b+
No. of Reads: 0 117870
No. of Writes: 2437 1291
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 57 FORGET
0.00 0.00 us 0.00 us 0.00 us 2144 RELEASE
0.00 0.00 us 0.00 us 0.00 us 1951 RELEASEDIR
1.74 51.00 us 29.00 us 73.00 us 2 READDIR
7.63 448.00 us 448.00 us 448.00 us 1 OPENDIR
8.31 488.00 us 488.00 us 488.00 us 1 GETXATTR
35.12 137.47 us 67.00 us 295.00 us 15 LOOKUP
47.21 92.40 us 23.00 us 695.00 us 30 INODELK
Duration: 15820 seconds
Data Read: 4919430656 bytes
Data Written: 6718739968 bytes
Interval 0 Stats:
Block Size: 1b+ 512b+ 1024b+
No. of Reads: 0 2064583 0
No. of Writes: 3 99280 2270930
Block Size: 2048b+ 4096b+ 8192b+
No. of Reads: 0 0 0
No. of Writes: 1699103 6563 5390
Block Size: 16384b+ 32768b+
No. of Reads: 0 117870
No. of Writes: 2437 1291
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 57 FORGET
0.00 0.00 us 0.00 us 0.00 us 2144 RELEASE
0.00 0.00 us 0.00 us 0.00 us 1951 RELEASEDIR
1.74 51.00 us 29.00 us 73.00 us 2 READDIR
7.63 448.00 us 448.00 us 448.00 us 1 OPENDIR
8.31 488.00 us 488.00 us 488.00 us 1 GETXATTR
35.12 137.47 us 67.00 us 295.00 us 15 LOOKUP
47.21 92.40 us 23.00 us 695.00 us 30 INODELK
Duration: 15820 seconds
Data Read: 4919430656 bytes
Data Written: 6718739968 bytes
Brick: 10.70.35.116:/rhs/brick2/disperse
----------------------------------------
Cumulative Stats:
Block Size: 1b+ 512b+ 1024b+
No. of Reads: 0 2070328 0
No. of Writes: 3 99280 2270930
Block Size: 2048b+ 4096b+ 8192b+
No. of Reads: 0 0 0
No. of Writes: 1699103 6563 5390
Block Size: 16384b+ 32768b+
No. of Reads: 0 116906
No. of Writes: 2437 1291
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 57 FORGET
0.00 0.00 us 0.00 us 0.00 us 2144 RELEASE
0.00 0.00 us 0.00 us 0.00 us 1951 RELEASEDIR
2.06 84.00 us 84.00 us 84.00 us 1 OPENDIR
3.60 73.50 us 67.00 us 80.00 us 2 READDIR
4.38 179.00 us 179.00 us 179.00 us 1 GETXATTR
42.42 57.77 us 39.00 us 97.00 us 30 INODELK
47.54 129.47 us 85.00 us 227.00 us 15 LOOKUP
Duration: 15819 seconds
Data Read: 4890783744 bytes
Data Written: 6718739968 bytes
Interval 0 Stats:
Block Size: 1b+ 512b+ 1024b+
No. of Reads: 0 2070328 0
No. of Writes: 3 99280 2270930
Block Size: 2048b+ 4096b+ 8192b+
No. of Reads: 0 0 0
No. of Writes: 1699103 6563 5390
Block Size: 16384b+ 32768b+
No. of Reads: 0 116906
No. of Writes: 2437 1291
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 57 FORGET
0.00 0.00 us 0.00 us 0.00 us 2144 RELEASE
0.00 0.00 us 0.00 us 0.00 us 1951 RELEASEDIR
2.06 84.00 us 84.00 us 84.00 us 1 OPENDIR
3.60 73.50 us 67.00 us 80.00 us 2 READDIR
4.38 179.00 us 179.00 us 179.00 us 1 GETXATTR
42.42 57.77 us 39.00 us 97.00 us 30 INODELK
47.54 129.47 us 85.00 us 227.00 us 15 LOOKUP
Duration: 15819 seconds
Data Read: 4890783744 bytes
Data Written: 6718739968 bytes
Brick: 10.70.35.116:/rhs/brick3/disperse
----------------------------------------
Cumulative Stats:
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 902198 0 0
No. of Writes: 70203 1693147 573317
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 0 0 0
No. of Writes: 3144 2175 656
Block Size: 32768b+
No. of Reads: 0
No. of Writes: 262
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 62 FORGET
0.00 0.00 us 0.00 us 0.00 us 1175 RELEASE
0.00 0.00 us 0.00 us 0.00 us 1957 RELEASEDIR
1.62 30.50 us 27.00 us 34.00 us 2 READDIR
1.73 65.00 us 65.00 us 65.00 us 1 OPENDIR
3.62 136.00 us 136.00 us 136.00 us 1 GETXATTR
42.87 107.40 us 74.00 us 156.00 us 15 LOOKUP
50.16 62.83 us 28.00 us 291.00 us 30 INODELK
Duration: 15819 seconds
Data Read: 461925376 bytes
Data Written: 3447035392 bytes
Interval 0 Stats:
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 902198 0 0
No. of Writes: 70203 1693147 573317
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 0 0 0
No. of Writes: 3144 2175 656
Block Size: 32768b+
No. of Reads: 0
No. of Writes: 262
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 62 FORGET
0.00 0.00 us 0.00 us 0.00 us 1175 RELEASE
0.00 0.00 us 0.00 us 0.00 us 1957 RELEASEDIR
1.62 30.50 us 27.00 us 34.00 us 2 READDIR
1.73 65.00 us 65.00 us 65.00 us 1 OPENDIR
3.62 136.00 us 136.00 us 136.00 us 1 GETXATTR
42.87 107.40 us 74.00 us 156.00 us 15 LOOKUP
50.16 62.83 us 28.00 us 291.00 us 30 INODELK
Duration: 15819 seconds
Data Read: 461925376 bytes
Data Written: 3447035392 bytes
Brick: 10.70.35.239:/rhs/brick3/disperse
----------------------------------------
Cumulative Stats:
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 902906 0 0
No. of Writes: 70203 1693147 573317
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 0 0 0
No. of Writes: 3144 2175 656
Block Size: 32768b+
No. of Reads: 0
No. of Writes: 262
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 62 FORGET
0.00 0.00 us 0.00 us 0.00 us 1175 RELEASE
0.00 0.00 us 0.00 us 0.00 us 1957 RELEASEDIR
1.82 67.00 us 67.00 us 67.00 us 1 OPENDIR
3.08 56.50 us 40.00 us 73.00 us 2 READDIR
3.76 138.00 us 138.00 us 138.00 us 1 GETXATTR
45.06 55.17 us 22.00 us 107.00 us 30 INODELK
46.28 113.33 us 75.00 us 180.00 us 15 LOOKUP
Duration: 15819 seconds
Data Read: 462287872 bytes
Data Written: 3447035392 bytes
Interval 0 Stats:
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 902906 0 0
No. of Writes: 70203 1693147 573317
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 0 0 0
No. of Writes: 3144 2175 656
Block Size: 32768b+
No. of Reads: 0
No. of Writes: 262
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 62 FORGET
0.00 0.00 us 0.00 us 0.00 us 1175 RELEASE
0.00 0.00 us 0.00 us 0.00 us 1957 RELEASEDIR
1.82 67.00 us 67.00 us 67.00 us 1 OPENDIR
3.08 56.50 us 40.00 us 73.00 us 2 READDIR
3.76 138.00 us 138.00 us 138.00 us 1 GETXATTR
45.06 55.17 us 22.00 us 107.00 us 30 INODELK
46.28 113.33 us 75.00 us 180.00 us 15 LOOKUP
Duration: 15819 seconds
Data Read: 462287872 bytes
Data Written: 3447035392 bytes
Brick: 10.70.35.239:/rhs/brick2/disperse
----------------------------------------
Cumulative Stats:
Block Size: 1b+ 512b+ 1024b+
No. of Reads: 0 2062666 0
No. of Writes: 3 99280 2270930
Block Size: 2048b+ 4096b+ 8192b+
No. of Reads: 0 0 0
No. of Writes: 1699103 6563 5390
Block Size: 16384b+ 32768b+
No. of Reads: 0 117681
No. of Writes: 2437 1291
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 57 FORGET
0.00 0.00 us 0.00 us 0.00 us 2144 RELEASE
0.00 0.00 us 0.00 us 0.00 us 1950 RELEASEDIR
2.00 38.50 us 36.00 us 41.00 us 2 READDIR
2.23 86.00 us 86.00 us 86.00 us 1 OPENDIR
4.51 174.00 us 174.00 us 174.00 us 1 GETXATTR
44.27 56.87 us 22.00 us 162.00 us 30 INODELK
46.99 120.73 us 60.00 us 234.00 us 15 LOOKUP
Duration: 15819 seconds
Data Read: 4912256000 bytes
Data Written: 6718739968 bytes
Interval 0 Stats:
Block Size: 1b+ 512b+ 1024b+
No. of Reads: 0 2062666 0
No. of Writes: 3 99280 2270930
Block Size: 2048b+ 4096b+ 8192b+
No. of Reads: 0 0 0
No. of Writes: 1699103 6563 5390
Block Size: 16384b+ 32768b+
No. of Reads: 0 117681
No. of Writes: 2437 1291
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 57 FORGET
0.00 0.00 us 0.00 us 0.00 us 2144 RELEASE
0.00 0.00 us 0.00 us 0.00 us 1950 RELEASEDIR
2.00 38.50 us 36.00 us 41.00 us 2 READDIR
2.23 86.00 us 86.00 us 86.00 us 1 OPENDIR
4.51 174.00 us 174.00 us 174.00 us 1 GETXATTR
44.27 56.87 us 22.00 us 162.00 us 30 INODELK
46.99 120.73 us 60.00 us 234.00 us 15 LOOKUP
Duration: 15819 seconds
Data Read: 4912256000 bytes
Data Written: 6718739968 bytes
Brick: 10.70.35.135:/rhs/brick3/disperse
----------------------------------------
Cumulative Stats:
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 902157 0 0
No. of Writes: 70203 1693147 573317
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 0 0 0
No. of Writes: 3144 2175 656
Block Size: 32768b+
No. of Reads: 0
No. of Writes: 262
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 62 FORGET
0.00 0.00 us 0.00 us 0.00 us 1175 RELEASE
0.00 0.00 us 0.00 us 0.00 us 1957 RELEASEDIR
1.63 69.00 us 69.00 us 69.00 us 1 OPENDIR
1.89 40.00 us 26.00 us 54.00 us 2 READDIR
5.69 241.00 us 241.00 us 241.00 us 1 GETXATTR
41.60 58.77 us 21.00 us 287.00 us 30 INODELK
49.20 139.00 us 78.00 us 276.00 us 15 LOOKUP
Duration: 15819 seconds
Data Read: 461904384 bytes
Data Written: 3447035392 bytes
Interval 0 Stats:
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 902157 0 0
No. of Writes: 70203 1693147 573317
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 0 0 0
No. of Writes: 3144 2175 656
Block Size: 32768b+
No. of Reads: 0
No. of Writes: 262
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 62 FORGET
0.00 0.00 us 0.00 us 0.00 us 1175 RELEASE
0.00 0.00 us 0.00 us 0.00 us 1957 RELEASEDIR
1.63 69.00 us 69.00 us 69.00 us 1 OPENDIR
1.89 40.00 us 26.00 us 54.00 us 2 READDIR
5.69 241.00 us 241.00 us 241.00 us 1 GETXATTR
41.60 58.77 us 21.00 us 287.00 us 30 INODELK
49.20 139.00 us 78.00 us 276.00 us 15 LOOKUP
Duration: 15819 seconds
Data Read: 461904384 bytes
Data Written: 3447035392 bytes
Brick: 10.70.35.135:/rhs/brick2/disperse
----------------------------------------
Cumulative Stats:
Block Size: 1b+ 512b+ 1024b+
No. of Reads: 0 2065363 0
No. of Writes: 3 99280 2270930
Block Size: 2048b+ 4096b+ 8192b+
No. of Reads: 0 0 0
No. of Writes: 1699103 6563 5390
Block Size: 16384b+ 32768b+
No. of Reads: 0 116863
No. of Writes: 2437 1291
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 57 FORGET
0.00 0.00 us 0.00 us 0.00 us 2144 RELEASE
0.00 0.00 us 0.00 us 0.00 us 1951 RELEASEDIR
1.93 36.00 us 25.00 us 47.00 us 2 READDIR
2.07 77.00 us 77.00 us 77.00 us 1 OPENDIR
5.64 210.00 us 210.00 us 210.00 us 1 GETXATTR
41.32 51.27 us 22.00 us 254.00 us 30 INODELK
49.03 121.67 us 60.00 us 386.00 us 15 LOOKUP
Duration: 15819 seconds
Data Read: 4886832640 bytes
Data Written: 6718739968 bytes
Interval 0 Stats:
Block Size: 1b+ 512b+ 1024b+
No. of Reads: 0 2065363 0
No. of Writes: 3 99280 2270930
Block Size: 2048b+ 4096b+ 8192b+
No. of Reads: 0 0 0
No. of Writes: 1699103 6563 5390
Block Size: 16384b+ 32768b+
No. of Reads: 0 116863
No. of Writes: 2437 1291
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 57 FORGET
0.00 0.00 us 0.00 us 0.00 us 2144 RELEASE
0.00 0.00 us 0.00 us 0.00 us 1951 RELEASEDIR
1.93 36.00 us 25.00 us 47.00 us 2 READDIR
2.07 77.00 us 77.00 us 77.00 us 1 OPENDIR
5.64 210.00 us 210.00 us 210.00 us 1 GETXATTR
41.32 51.27 us 22.00 us 254.00 us 30 INODELK
49.03 121.67 us 60.00 us 386.00 us 15 LOOKUP
Duration: 15819 seconds
Data Read: 4886832640 bytes
Data Written: 6718739968 bytes
Brick: 10.70.35.8:/rhs/brick2/disperse
--------------------------------------
Cumulative Stats:
Block Size: 1b+ 512b+ 1024b+
No. of Reads: 0 2061857 0
No. of Writes: 3 99280 2270930
Block Size: 2048b+ 4096b+ 8192b+
No. of Reads: 0 0 0
No. of Writes: 1699103 6563 5390
Block Size: 16384b+ 32768b+
No. of Reads: 0 116019
No. of Writes: 2437 1291
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 57 FORGET
0.00 0.00 us 0.00 us 0.00 us 2144 RELEASE
0.00 0.00 us 0.00 us 0.00 us 1951 RELEASEDIR
2.52 111.00 us 111.00 us 111.00 us 1 OPENDIR
3.57 78.50 us 73.00 us 84.00 us 2 READDIR
7.43 327.00 us 327.00 us 327.00 us 1 GETXATTR
42.72 62.70 us 23.00 us 172.00 us 30 INODELK
43.77 128.47 us 60.00 us 274.00 us 15 LOOKUP
Duration: 15819 seconds
Data Read: 4857381376 bytes
Data Written: 6718739968 bytes
Interval 0 Stats:
Block Size: 1b+ 512b+ 1024b+
No. of Reads: 0 2061857 0
No. of Writes: 3 99280 2270930
Block Size: 2048b+ 4096b+ 8192b+
No. of Reads: 0 0 0
No. of Writes: 1699103 6563 5390
Block Size: 16384b+ 32768b+
No. of Reads: 0 116019
No. of Writes: 2437 1291
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 57 FORGET
0.00 0.00 us 0.00 us 0.00 us 2144 RELEASE
0.00 0.00 us 0.00 us 0.00 us 1951 RELEASEDIR
2.52 111.00 us 111.00 us 111.00 us 1 OPENDIR
3.57 78.50 us 73.00 us 84.00 us 2 READDIR
7.43 327.00 us 327.00 us 327.00 us 1 GETXATTR
42.72 62.70 us 23.00 us 172.00 us 30 INODELK
43.77 128.47 us 60.00 us 274.00 us 15 LOOKUP
Duration: 15819 seconds
Data Read: 4857381376 bytes
Data Written: 6718739968 bytes
Brick: 10.70.35.8:/rhs/brick3/disperse
--------------------------------------
Cumulative Stats:
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 900576 0 0
No. of Writes: 70203 1693147 573317
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 0 0 0
No. of Writes: 3144 2175 656
Block Size: 32768b+
No. of Reads: 0
No. of Writes: 262
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 62 FORGET
0.00 0.00 us 0.00 us 0.00 us 1175 RELEASE
0.00 0.00 us 0.00 us 0.00 us 1957 RELEASEDIR
1.45 28.00 us 26.00 us 30.00 us 2 READDIR
1.66 64.00 us 64.00 us 64.00 us 1 OPENDIR
3.80 147.00 us 147.00 us 147.00 us 1 GETXATTR
45.51 58.67 us 22.00 us 176.00 us 30 INODELK
47.58 122.67 us 67.00 us 249.00 us 15 LOOKUP
Duration: 15818 seconds
Data Read: 461094912 bytes
Data Written: 3447035392 bytes
Interval 0 Stats:
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 900576 0 0
No. of Writes: 70203 1693147 573317
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 0 0 0
No. of Writes: 3144 2175 656
Block Size: 32768b+
No. of Reads: 0
No. of Writes: 262
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 62 FORGET
0.00 0.00 us 0.00 us 0.00 us 1175 RELEASE
0.00 0.00 us 0.00 us 0.00 us 1957 RELEASEDIR
1.45 28.00 us 26.00 us 30.00 us 2 READDIR
1.66 64.00 us 64.00 us 64.00 us 1 OPENDIR
3.80 147.00 us 147.00 us 147.00 us 1 GETXATTR
45.51 58.67 us 22.00 us 176.00 us 30 INODELK
47.58 122.67 us 67.00 us 249.00 us 15 LOOKUP
Duration: 15818 seconds
Data Read: 461094912 bytes
Data Written: 3447035392 bytes
Brick: 10.70.35.196:/rhs/brick2/disperse
----------------------------------------
Cumulative Stats:
Block Size: 1b+ 512b+ 1024b+
No. of Reads: 0 2053415 0
No. of Writes: 3 99280 2270930
Block Size: 2048b+ 4096b+ 8192b+
No. of Reads: 0 0 0
No. of Writes: 1699103 6563 5390
Block Size: 16384b+ 32768b+
No. of Reads: 0 117801
No. of Writes: 2437 1291
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 57 FORGET
0.00 0.00 us 0.00 us 0.00 us 2144 RELEASE
0.00 0.00 us 0.00 us 0.00 us 1958 RELEASEDIR
1.66 39.50 us 9.00 us 70.00 us 2 OPENDIR
9.25 147.00 us 52.00 us 246.00 us 3 GETXATTR
17.71 211.00 us 94.00 us 389.00 us 4 READDIR
33.88 53.83 us 25.00 us 150.00 us 30 INODELK
37.51 111.75 us 61.00 us 258.00 us 16 LOOKUP
Duration: 15819 seconds
Data Read: 4911451648 bytes
Data Written: 6718739968 bytes
Interval 0 Stats:
Block Size: 1b+ 512b+ 1024b+
No. of Reads: 0 2053415 0
No. of Writes: 3 99280 2270930
Block Size: 2048b+ 4096b+ 8192b+
No. of Reads: 0 0 0
No. of Writes: 1699103 6563 5390
Block Size: 16384b+ 32768b+
No. of Reads: 0 117801
No. of Writes: 2437 1291
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 57 FORGET
0.00 0.00 us 0.00 us 0.00 us 2144 RELEASE
0.00 0.00 us 0.00 us 0.00 us 1958 RELEASEDIR
1.66 39.50 us 9.00 us 70.00 us 2 OPENDIR
9.25 147.00 us 52.00 us 246.00 us 3 GETXATTR
17.71 211.00 us 94.00 us 389.00 us 4 READDIR
33.88 53.83 us 25.00 us 150.00 us 30 INODELK
37.51 111.75 us 61.00 us 258.00 us 16 LOOKUP
Duration: 15819 seconds
Data Read: 4911451648 bytes
Data Written: 6718739968 bytes
Brick: 10.70.35.196:/rhs/brick3/disperse
----------------------------------------
Cumulative Stats:
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 900507 0 0
No. of Writes: 70203 1693147 573317
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 0 0 0
No. of Writes: 3144 2175 656
Block Size: 32768b+
No. of Reads: 0
No. of Writes: 262
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 62 FORGET
0.00 0.00 us 0.00 us 0.00 us 1175 RELEASE
0.00 0.00 us 0.00 us 0.00 us 1964 RELEASEDIR
1.52 35.00 us 2.00 us 68.00 us 2 OPENDIR
7.44 114.33 us 63.00 us 142.00 us 3 GETXATTR
15.80 182.25 us 38.00 us 426.00 us 4 READDIR
35.46 54.53 us 29.00 us 141.00 us 30 INODELK
39.78 114.69 us 81.00 us 198.00 us 16 LOOKUP
Duration: 15818 seconds
Data Read: 461059584 bytes
Data Written: 3447035392 bytes
Interval 0 Stats:
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 900507 0 0
No. of Writes: 70203 1693147 573317
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 0 0 0
No. of Writes: 3144 2175 656
Block Size: 32768b+
No. of Reads: 0
No. of Writes: 262
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 62 FORGET
0.00 0.00 us 0.00 us 0.00 us 1175 RELEASE
0.00 0.00 us 0.00 us 0.00 us 1964 RELEASEDIR
1.52 35.00 us 2.00 us 68.00 us 2 OPENDIR
7.44 114.33 us 63.00 us 142.00 us 3 GETXATTR
15.80 182.25 us 38.00 us 426.00 us 4 READDIR
35.46 54.53 us 29.00 us 141.00 us 30 INODELK
39.78 114.69 us 81.00 us 198.00 us 16 LOOKUP
Duration: 15818 seconds
Data Read: 461059584 bytes
Data Written: 3447035392 bytes
upstream mainline patch http://review.gluster.org/16006 posted for review. (In reply to Atin Mukherjee from comment #9) > upstream mainline patch http://review.gluster.org/16006 posted for review. Correction, this is a 3.9 patch, mainline patch is http://review.gluster.org/#/c/16007 downstream patch : https://code.engineering.redhat.com/gerrit/92048 on_qa validation
ran the below case and found the memory consumption constant. Hence marking as passed.
validated on 3.8.4-8
1.create a 2x(4+2) ec volume
2.mount on two fuse clients
3.keep tracking the top and free comamnd for the fuse mount(where you will be doing dd)
4. from one client dd a file of size 2Gb and as soon as this starts, from 2nd client issue a truncate of size 5GB
I didn't notice any spike
client1 where dd was happening:
[root@rhs-client45 ~]# for i in {1..10000};do top -n 1 -b|egrep "RES|6967";sleep 30;done
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6967 root 20 0 649480 51384 3136 S 0.0 0.3 0:00.07 gluster+
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6967 root 20 0 649480 51384 3136 S 0.0 0.3 0:00.07 gluster+
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6967 root 20 0 649480 51824 3136 S 0.0 0.3 0:00.07 gluster+
==========>started dd
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6967 root 20 0 649480 57104 3824 S 93.8 0.4 0:28.92 gluster+
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6967 root 20 0 649480 59232 3840 S 106.2 0.4 0:58.07 gluster+
====> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6967 root 20 0 649480 59232 3840 S 93.8 0.4 1:27.33 gluster+
======>started trucate
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6967 root 20 0 649480 59232 3840 S 93.8 0.4 1:53.77 gluster+
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6967 root 20 0 649480 59236 3840 S 100.0 0.4 2:23.98 gluster+
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6967 root 20 0 649480 59244 3840 S 100.0 0.4 2:53.03 gluster+
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6967 root 20 0 649480 59244 3840 S 106.2 0.4 3:22.03 gluster+
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6967 root 20 0 649480 59244 3840 S 106.2 0.4 3:50.41 gluster+
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6967 root 20 0 649480 59244 3840 S 93.8 0.4 4:19.49 gluster+
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6967 root 20 0 649480 59244 3840 S 93.8 0.4 4:47.89 gluster+
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6967 root 20 0 649480 59244 3840 S 0.0 0.4 5:09.92 gluster+
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6967 root 20 0 848144 60228 3904 S 68.8 0.4 5:17.60 gluster+
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6967 root 20 0 848144 60248 3908 S 43.8 0.4 5:34.62 gluster+
=======================================
client2 where truncate was issued
[root@rhs-client24 ~]# for i in {1..10000};do top -n 1 -b|egrep "RES|6428";sleep 30;done
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6428 root 20 0 638m 46m 3220 S 0.0 0.3 0:00.12 glusterfs
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6428 root 20 0 638m 47m 3220 S 0.0 0.3 0:00.13 glusterfs
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6428 root 20 0 638m 47m 3220 S 0.0 0.3 0:00.13 glusterfs
==========>started dd
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6428 root 20 0 702m 48m 3484 S 0.0 0.3 0:00.16 glusterfs
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6428 root 20 0 702m 48m 3484 S 0.0 0.3 0:00.16 glusterfs
======>started trucate
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6428 root 20 0 702m 50m 3760 S 0.0 0.3 0:00.17 glusterfs
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6428 root 20 0 702m 50m 3760 S 0.0 0.3 0:00.18 glusterfs
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6428 root 20 0 702m 50m 3760 S 0.0 0.3 0:00.18 glusterfs
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6428 root 20 0 702m 50m 3760 S 0.0 0.3 0:00.18 glusterfs
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6428 root 20 0 702m 50m 3760 S 0.0 0.3 0:00.18 glusterfs
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6428 root 20 0 702m 50m 3760 S 0.0 0.3 0:00.18 glusterfs
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6428 root 20 0 702m 50m 3760 S 0.0 0.3 0:00.18 glusterfs
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6428 root 20 0 702m 50m 3760 S 0.0 0.3 0:00.18 glusterfs
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6428 root 20 0 702m 50m 3760 S 0.0 0.3 0:00.20 glusterfs
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6428 root 20 0 702m 54m 3920 S 30.8 0.3 0:00.49 glusterfs
===============================
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6428 root 20 0 702m 55m 3920 S 36.5 0.3 0:11.82 glusterfs
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2017-0486.html |