Description of problem: ------------------------ The problem is fairly reproducible on EC 96*(4+2) and on a small 2*2 volume as well,with millions and millions of files. Ran find from a single mount. Karan could repro this on 2*2 as well during rm. The core doesn't show much though : [root@gqac006 tmp]# gdb /usr/bin/glusterfs core.dump.PID\=29290UID\=0 GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-94.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... /usr/bin/glusterfs: No such file or directory. [New LWP 2479] [New LWP 29459] [New LWP 29291] [New LWP 29294] [New LWP 29290] [New LWP 29304] [New LWP 29295] [New LWP 2476] [New LWP 29303] [New LWP 29300] [New LWP 29292] [New LWP 29298] [New LWP 29299] [New LWP 29293] [New LWP 2477] [New LWP 29296] Missing separate debuginfo for the main executable file Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/9e/208d45be8ce12ba3da094d8241bdfa0e627fd0 Core was generated by `/usr/sbin/glusterfs --volfile-server=gqas001.sbu.lab.eng.bos.redhat.com --volfi'. Program terminated with signal 11, Segmentation fault. #0 0x00007f44fa08ec1b in ?? () (gdb) bt Python Exception <class 'gdb.MemoryError'> Cannot access memory at address 0x7f44ec6957c0: (gdb) Version-Release number of selected component (if applicable): ------------------------------------------------------------- 3.8.4-25 How reproducible: ----------------- Fairly Actual results: --------------- client process crashes. Expected results: ----------------- No crashes. Additional info: ---------------- *For DR * [root@gqas004 ~]# gluster v info Volume Name: testvol Type: Distributed-Replicate Volume ID: f1259ca8-d87e-4461-8441-84996d2ccda5 Status: Started Snapshot Count: 0 Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: gqas001.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick0 Brick2: gqas004.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick1 Brick3: gqas010.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick2 Brick4: gqas012.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick3 Options Reconfigured: client.event-threads: 4 server.event-threads: 4 cluster.lookup-optimize: on network.inode-lru-limit: 90000 performance.md-cache-timeout: 600 performance.cache-invalidation: on features.cache-invalidation-timeout: 600 features.cache-invalidation: on server.allow-insecure: on performance.stat-prefetch: on transport.address-family: inet nfs.disable: on [root@gqas004 ~]# For *EC* : [root@gqas013 ~]# gluster v info Volume Name: butcher Type: Distributed-Disperse Volume ID: 51e75bff-f8b3-4407-acd8-4d09ca136964 Status: Started Snapshot Count: 0 Number of Bricks: 96 x (4 + 2) = 576 Transport-type: tcp Bricks: Brick1: gqas013:/bricks1/1 Brick2: gqas014:/bricks1/1 Brick3: gqas015:/bricks1/1 Brick4: gqas005:/bricks1/1 Brick5: gqas006:/bricks1/1 Brick6: gqas008:/bricks1/1 Brick7: gqas013:/bricks1/2 Brick8: gqas014:/bricks1/2 Brick9: gqas015:/bricks1/2 Brick10: gqas005:/bricks1/2 Brick11: gqas006:/bricks1/2 Brick12: gqas008:/bricks1/2 Brick13: gqas013:/bricks1/3 Brick14: gqas014:/bricks1/3 Brick15: gqas015:/bricks1/3 Brick16: gqas005:/bricks1/3 Brick17: gqas006:/bricks1/3 Brick18: gqas008:/bricks1/3 Brick19: gqas013:/bricks1/4 Brick20: gqas014:/bricks1/4 Brick21: gqas015:/bricks1/4 Brick22: gqas005:/bricks1/4 Brick23: gqas006:/bricks1/4 Brick24: gqas008:/bricks1/4 Brick25: gqas013:/bricks1/5 Brick26: gqas014:/bricks1/5 Brick27: gqas015:/bricks1/5 Brick28: gqas005:/bricks1/5 Brick29: gqas006:/bricks1/5 Brick30: gqas008:/bricks1/5 Brick31: gqas013:/bricks1/6 Brick32: gqas014:/bricks1/6 Brick33: gqas015:/bricks1/6 Brick34: gqas005:/bricks1/6 Brick35: gqas006:/bricks1/6 Brick36: gqas008:/bricks1/6 Brick37: gqas013:/bricks1/7 Brick38: gqas014:/bricks1/7 Brick39: gqas015:/bricks1/7 Brick40: gqas005:/bricks1/7 Brick41: gqas006:/bricks1/7 Brick42: gqas008:/bricks1/7 Brick43: gqas013:/bricks1/8 Brick44: gqas014:/bricks1/8 Brick45: gqas015:/bricks1/8 Brick46: gqas005:/bricks1/8 Brick47: gqas006:/bricks1/8 Brick48: gqas008:/bricks1/8 Brick49: gqas013:/bricks2/1 Brick50: gqas014:/bricks2/1 Brick51: gqas015:/bricks2/1 Brick52: gqas005:/bricks2/1 Brick53: gqas006:/bricks2/1 Brick54: gqas008:/bricks2/1 Brick55: gqas013:/bricks2/2 Brick56: gqas014:/bricks2/2 Brick57: gqas015:/bricks2/2 Brick58: gqas005:/bricks2/2 Brick59: gqas006:/bricks2/2 Brick60: gqas008:/bricks2/2 Brick61: gqas013:/bricks2/3 Brick62: gqas014:/bricks2/3 Brick63: gqas015:/bricks2/3 Brick64: gqas005:/bricks2/3 Brick65: gqas006:/bricks2/3 Brick66: gqas008:/bricks2/3 Brick67: gqas013:/bricks2/4 Brick68: gqas014:/bricks2/4 Brick69: gqas015:/bricks2/4 Brick70: gqas005:/bricks2/4 Brick71: gqas006:/bricks2/4 Brick72: gqas008:/bricks2/4 Brick73: gqas013:/bricks2/5 Brick74: gqas014:/bricks2/5 Brick75: gqas015:/bricks2/5 Brick76: gqas005:/bricks2/5 Brick77: gqas006:/bricks2/5 Brick78: gqas008:/bricks2/5 Brick79: gqas013:/bricks2/6 Brick80: gqas014:/bricks2/6 Brick81: gqas015:/bricks2/6 Brick82: gqas005:/bricks2/6 Brick83: gqas006:/bricks2/6 Brick84: gqas008:/bricks2/6 Brick85: gqas013:/bricks2/7 Brick86: gqas014:/bricks2/7 Brick87: gqas015:/bricks2/7 Brick88: gqas005:/bricks2/7 Brick89: gqas006:/bricks2/7 Brick90: gqas008:/bricks2/7 Brick91: gqas013:/bricks2/8 Brick92: gqas014:/bricks2/8 Brick93: gqas015:/bricks2/8 Brick94: gqas005:/bricks2/8 Brick95: gqas006:/bricks2/8 Brick96: gqas008:/bricks2/8 Brick97: gqas013:/bricks3/1 Brick98: gqas014:/bricks3/1 Brick99: gqas015:/bricks3/1 Brick100: gqas005:/bricks3/1 Brick101: gqas006:/bricks3/1 Brick102: gqas008:/bricks3/1 Brick103: gqas013:/bricks3/2 Brick104: gqas014:/bricks3/2 Brick105: gqas015:/bricks3/2 Brick106: gqas005:/bricks3/2 Brick107: gqas006:/bricks3/2 Brick108: gqas008:/bricks3/2 Brick109: gqas013:/bricks3/3 Brick110: gqas014:/bricks3/3 Brick111: gqas015:/bricks3/3 Brick112: gqas005:/bricks3/3 Brick113: gqas006:/bricks3/3 Brick114: gqas008:/bricks3/3 Brick115: gqas013:/bricks3/4 Brick116: gqas014:/bricks3/4 Brick117: gqas015:/bricks3/4 Brick118: gqas005:/bricks3/4 Brick119: gqas006:/bricks3/4 Brick120: gqas008:/bricks3/4 Brick121: gqas013:/bricks3/5 Brick122: gqas014:/bricks3/5 Brick123: gqas015:/bricks3/5 Brick124: gqas005:/bricks3/5 Brick125: gqas006:/bricks3/5 Brick126: gqas008:/bricks3/5 Brick127: gqas013:/bricks3/6 Brick128: gqas014:/bricks3/6 Brick129: gqas015:/bricks3/6 Brick130: gqas005:/bricks3/6 Brick131: gqas006:/bricks3/6 Brick132: gqas008:/bricks3/6 Brick133: gqas013:/bricks3/7 Brick134: gqas014:/bricks3/7 Brick135: gqas015:/bricks3/7 Brick136: gqas005:/bricks3/7 Brick137: gqas006:/bricks3/7 Brick138: gqas008:/bricks3/7 Brick139: gqas013:/bricks3/8 Brick140: gqas014:/bricks3/8 Brick141: gqas015:/bricks3/8 Brick142: gqas005:/bricks3/8 Brick143: gqas006:/bricks3/8 Brick144: gqas008:/bricks3/8 Brick145: gqas013:/bricks4/1 Brick146: gqas014:/bricks4/1 Brick147: gqas015:/bricks4/1 Brick148: gqas005:/bricks4/1 Brick149: gqas006:/bricks4/1 Brick150: gqas008:/bricks4/1 Brick151: gqas013:/bricks4/2 Brick152: gqas014:/bricks4/2 Brick153: gqas015:/bricks4/2 Brick154: gqas005:/bricks4/2 Brick155: gqas006:/bricks4/2 Brick156: gqas008:/bricks4/2 Brick157: gqas013:/bricks4/3 Brick158: gqas014:/bricks4/3 Brick159: gqas015:/bricks4/3 Brick160: gqas005:/bricks4/3 Brick161: gqas006:/bricks4/3 Brick162: gqas008:/bricks4/3 Brick163: gqas013:/bricks4/4 Brick164: gqas014:/bricks4/4 Brick165: gqas015:/bricks4/4 Brick166: gqas005:/bricks4/4 Brick167: gqas006:/bricks4/4 Brick168: gqas008:/bricks4/4 Brick169: gqas013:/bricks4/5 Brick170: gqas014:/bricks4/5 Brick171: gqas015:/bricks4/5 Brick172: gqas005:/bricks4/5 Brick173: gqas006:/bricks4/5 Brick174: gqas008:/bricks4/5 Brick175: gqas013:/bricks4/6 Brick176: gqas014:/bricks4/6 Brick177: gqas015:/bricks4/6 Brick178: gqas005:/bricks4/6 Brick179: gqas006:/bricks4/6 Brick180: gqas008:/bricks4/6 Brick181: gqas013:/bricks4/7 Brick182: gqas014:/bricks4/7 Brick183: gqas015:/bricks4/7 Brick184: gqas005:/bricks4/7 Brick185: gqas006:/bricks4/7 Brick186: gqas008:/bricks4/7 Brick187: gqas013:/bricks4/8 Brick188: gqas014:/bricks4/8 Brick189: gqas015:/bricks4/8 Brick190: gqas005:/bricks4/8 Brick191: gqas006:/bricks4/8 Brick192: gqas008:/bricks4/8 Brick193: gqas013:/bricks5/1 Brick194: gqas014:/bricks5/1 Brick195: gqas015:/bricks5/1 Brick196: gqas005:/bricks5/1 Brick197: gqas006:/bricks5/1 Brick198: gqas008:/bricks5/1 Brick199: gqas013:/bricks5/2 Brick200: gqas014:/bricks5/2 Brick201: gqas015:/bricks5/2 Brick202: gqas005:/bricks5/2 Brick203: gqas006:/bricks5/2 Brick204: gqas008:/bricks5/2 Brick205: gqas013:/bricks5/3 Brick206: gqas014:/bricks5/3 Brick207: gqas015:/bricks5/3 Brick208: gqas005:/bricks5/3 Brick209: gqas006:/bricks5/3 Brick210: gqas008:/bricks5/3 Brick211: gqas013:/bricks5/4 Brick212: gqas014:/bricks5/4 Brick213: gqas015:/bricks5/4 Brick214: gqas005:/bricks5/4 Brick215: gqas006:/bricks5/4 Brick216: gqas008:/bricks5/4 Brick217: gqas013:/bricks5/5 Brick218: gqas014:/bricks5/5 Brick219: gqas015:/bricks5/5 Brick220: gqas005:/bricks5/5 Brick221: gqas006:/bricks5/5 Brick222: gqas008:/bricks5/5 Brick223: gqas013:/bricks5/6 Brick224: gqas014:/bricks5/6 Brick225: gqas015:/bricks5/6 Brick226: gqas005:/bricks5/6 Brick227: gqas006:/bricks5/6 Brick228: gqas008:/bricks5/6 Brick229: gqas013:/bricks5/7 Brick230: gqas014:/bricks5/7 Brick231: gqas015:/bricks5/7 Brick232: gqas005:/bricks5/7 Brick233: gqas006:/bricks5/7 Brick234: gqas008:/bricks5/7 Brick235: gqas013:/bricks5/8 Brick236: gqas014:/bricks5/8 Brick237: gqas015:/bricks5/8 Brick238: gqas005:/bricks5/8 Brick239: gqas006:/bricks5/8 Brick240: gqas008:/bricks5/8 Brick241: gqas013:/bricks6/1 Brick242: gqas014:/bricks6/1 Brick243: gqas015:/bricks6/1 Brick244: gqas005:/bricks6/1 Brick245: gqas006:/bricks6/1 Brick246: gqas008:/bricks6/1 Brick247: gqas013:/bricks6/2 Brick248: gqas014:/bricks6/2 Brick249: gqas015:/bricks6/2 Brick250: gqas005:/bricks6/2 Brick251: gqas006:/bricks6/2 Brick252: gqas008:/bricks6/2 Brick253: gqas013:/bricks6/3 Brick254: gqas014:/bricks6/3 Brick255: gqas015:/bricks6/3 Brick256: gqas005:/bricks6/3 Brick257: gqas006:/bricks6/3 Brick258: gqas008:/bricks6/3 Brick259: gqas013:/bricks6/4 Brick260: gqas014:/bricks6/4 Brick261: gqas015:/bricks6/4 Brick262: gqas005:/bricks6/4 Brick263: gqas006:/bricks6/4 Brick264: gqas008:/bricks6/4 Brick265: gqas013:/bricks6/5 Brick266: gqas014:/bricks6/5 Brick267: gqas015:/bricks6/5 Brick268: gqas005:/bricks6/5 Brick269: gqas006:/bricks6/5 Brick270: gqas008:/bricks6/5 Brick271: gqas013:/bricks6/6 Brick272: gqas014:/bricks6/6 Brick273: gqas015:/bricks6/6 Brick274: gqas005:/bricks6/6 Brick275: gqas006:/bricks6/6 Brick276: gqas008:/bricks6/6 Brick277: gqas013:/bricks6/7 Brick278: gqas014:/bricks6/7 Brick279: gqas015:/bricks6/7 Brick280: gqas005:/bricks6/7 Brick281: gqas006:/bricks6/7 Brick282: gqas008:/bricks6/7 Brick283: gqas013:/bricks6/8 Brick284: gqas014:/bricks6/8 Brick285: gqas015:/bricks6/8 Brick286: gqas005:/bricks6/8 Brick287: gqas006:/bricks6/8 Brick288: gqas008:/bricks6/8 Brick289: gqas013:/bricks7/1 Brick290: gqas014:/bricks7/1 Brick291: gqas015:/bricks7/1 Brick292: gqas005:/bricks7/1 Brick293: gqas006:/bricks7/1 Brick294: gqas008:/bricks7/1 Brick295: gqas013:/bricks7/2 Brick296: gqas014:/bricks7/2 Brick297: gqas015:/bricks7/2 Brick298: gqas005:/bricks7/2 Brick299: gqas006:/bricks7/2 Brick300: gqas008:/bricks7/2 Brick301: gqas013:/bricks7/3 Brick302: gqas014:/bricks7/3 Brick303: gqas015:/bricks7/3 Brick304: gqas005:/bricks7/3 Brick305: gqas006:/bricks7/3 Brick306: gqas008:/bricks7/3 Brick307: gqas013:/bricks7/4 Brick308: gqas014:/bricks7/4 Brick309: gqas015:/bricks7/4 Brick310: gqas005:/bricks7/4 Brick311: gqas006:/bricks7/4 Brick312: gqas008:/bricks7/4 Brick313: gqas013:/bricks7/5 Brick314: gqas014:/bricks7/5 Brick315: gqas015:/bricks7/5 Brick316: gqas005:/bricks7/5 Brick317: gqas006:/bricks7/5 Brick318: gqas008:/bricks7/5 Brick319: gqas013:/bricks7/6 Brick320: gqas014:/bricks7/6 Brick321: gqas015:/bricks7/6 Brick322: gqas005:/bricks7/6 Brick323: gqas006:/bricks7/6 Brick324: gqas008:/bricks7/6 Brick325: gqas013:/bricks7/7 Brick326: gqas014:/bricks7/7 Brick327: gqas015:/bricks7/7 Brick328: gqas005:/bricks7/7 Brick329: gqas006:/bricks7/7 Brick330: gqas008:/bricks7/7 Brick331: gqas013:/bricks7/8 Brick332: gqas014:/bricks7/8 Brick333: gqas015:/bricks7/8 Brick334: gqas005:/bricks7/8 Brick335: gqas006:/bricks7/8 Brick336: gqas008:/bricks7/8 Brick337: gqas013:/bricks8/1 Brick338: gqas014:/bricks8/1 Brick339: gqas015:/bricks8/1 Brick340: gqas005:/bricks8/1 Brick341: gqas006:/bricks8/1 Brick342: gqas008:/bricks8/1 Brick343: gqas013:/bricks8/2 Brick344: gqas014:/bricks8/2 Brick345: gqas015:/bricks8/2 Brick346: gqas005:/bricks8/2 Brick347: gqas006:/bricks8/2 Brick348: gqas008:/bricks8/2 Brick349: gqas013:/bricks8/3 Brick350: gqas014:/bricks8/3 Brick351: gqas015:/bricks8/3 Brick352: gqas005:/bricks8/3 Brick353: gqas006:/bricks8/3 Brick354: gqas008:/bricks8/3 Brick355: gqas013:/bricks8/4 Brick356: gqas014:/bricks8/4 Brick357: gqas015:/bricks8/4 Brick358: gqas005:/bricks8/4 Brick359: gqas006:/bricks8/4 Brick360: gqas008:/bricks8/4 Brick361: gqas013:/bricks8/5 Brick362: gqas014:/bricks8/5 Brick363: gqas015:/bricks8/5 Brick364: gqas005:/bricks8/5 Brick365: gqas006:/bricks8/5 Brick366: gqas008:/bricks8/5 Brick367: gqas013:/bricks8/6 Brick368: gqas014:/bricks8/6 Brick369: gqas015:/bricks8/6 Brick370: gqas005:/bricks8/6 Brick371: gqas006:/bricks8/6 Brick372: gqas008:/bricks8/6 Brick373: gqas013:/bricks8/7 Brick374: gqas014:/bricks8/7 Brick375: gqas015:/bricks8/7 Brick376: gqas005:/bricks8/7 Brick377: gqas006:/bricks8/7 Brick378: gqas008:/bricks8/7 Brick379: gqas013:/bricks8/8 Brick380: gqas014:/bricks8/8 Brick381: gqas015:/bricks8/8 Brick382: gqas005:/bricks8/8 Brick383: gqas006:/bricks8/8 Brick384: gqas008:/bricks8/8 Brick385: gqas013:/bricks9/1 Brick386: gqas014:/bricks9/1 Brick387: gqas015:/bricks9/1 Brick388: gqas005:/bricks9/1 Brick389: gqas006:/bricks9/1 Brick390: gqas008:/bricks9/1 Brick391: gqas013:/bricks9/2 Brick392: gqas014:/bricks9/2 Brick393: gqas015:/bricks9/2 Brick394: gqas005:/bricks9/2 Brick395: gqas006:/bricks9/2 Brick396: gqas008:/bricks9/2 Brick397: gqas013:/bricks9/3 Brick398: gqas014:/bricks9/3 Brick399: gqas015:/bricks9/3 Brick400: gqas005:/bricks9/3 Brick401: gqas006:/bricks9/3 Brick402: gqas008:/bricks9/3 Brick403: gqas013:/bricks9/4 Brick404: gqas014:/bricks9/4 Brick405: gqas015:/bricks9/4 Brick406: gqas005:/bricks9/4 Brick407: gqas006:/bricks9/4 Brick408: gqas008:/bricks9/4 Brick409: gqas013:/bricks9/5 Brick410: gqas014:/bricks9/5 Brick411: gqas015:/bricks9/5 Brick412: gqas005:/bricks9/5 Brick413: gqas006:/bricks9/5 Brick414: gqas008:/bricks9/5 Brick415: gqas013:/bricks9/6 Brick416: gqas014:/bricks9/6 Brick417: gqas015:/bricks9/6 Brick418: gqas005:/bricks9/6 Brick419: gqas006:/bricks9/6 Brick420: gqas008:/bricks9/6 Brick421: gqas013:/bricks9/7 Brick422: gqas014:/bricks9/7 Brick423: gqas015:/bricks9/7 Brick424: gqas005:/bricks9/7 Brick425: gqas006:/bricks9/7 Brick426: gqas008:/bricks9/7 Brick427: gqas013:/bricks9/8 Brick428: gqas014:/bricks9/8 Brick429: gqas015:/bricks9/8 Brick430: gqas005:/bricks9/8 Brick431: gqas006:/bricks9/8 Brick432: gqas008:/bricks9/8 Brick433: gqas013:/bricks10/1 Brick434: gqas014:/bricks10/1 Brick435: gqas015:/bricks10/1 Brick436: gqas005:/bricks10/1 Brick437: gqas006:/bricks10/1 Brick438: gqas008:/bricks10/1 Brick439: gqas013:/bricks10/2 Brick440: gqas014:/bricks10/2 Brick441: gqas015:/bricks10/2 Brick442: gqas005:/bricks10/2 Brick443: gqas006:/bricks10/2 Brick444: gqas008:/bricks10/2 Brick445: gqas013:/bricks10/3 Brick446: gqas014:/bricks10/3 Brick447: gqas015:/bricks10/3 Brick448: gqas005:/bricks10/3 Brick449: gqas006:/bricks10/3 Brick450: gqas008:/bricks10/3 Brick451: gqas013:/bricks10/4 Brick452: gqas014:/bricks10/4 Brick453: gqas015:/bricks10/4 Brick454: gqas005:/bricks10/4 Brick455: gqas006:/bricks10/4 Brick456: gqas008:/bricks10/4 Brick457: gqas013:/bricks10/5 Brick458: gqas014:/bricks10/5 Brick459: gqas015:/bricks10/5 Brick460: gqas005:/bricks10/5 Brick461: gqas006:/bricks10/5 Brick462: gqas008:/bricks10/5 Brick463: gqas013:/bricks10/6 Brick464: gqas014:/bricks10/6 Brick465: gqas015:/bricks10/6 Brick466: gqas005:/bricks10/6 Brick467: gqas006:/bricks10/6 Brick468: gqas008:/bricks10/6 Brick469: gqas013:/bricks10/7 Brick470: gqas014:/bricks10/7 Brick471: gqas015:/bricks10/7 Brick472: gqas005:/bricks10/7 Brick473: gqas006:/bricks10/7 Brick474: gqas008:/bricks10/7 Brick475: gqas013:/bricks10/8 Brick476: gqas014:/bricks10/8 Brick477: gqas015:/bricks10/8 Brick478: gqas005:/bricks10/8 Brick479: gqas006:/bricks10/8 Brick480: gqas008:/bricks10/8 Brick481: gqas013:/bricks11/1 Brick482: gqas014:/bricks11/1 Brick483: gqas015:/bricks11/1 Brick484: gqas005:/bricks11/1 Brick485: gqas006:/bricks11/1 Brick486: gqas008:/bricks11/1 Brick487: gqas013:/bricks11/2 Brick488: gqas014:/bricks11/2 Brick489: gqas015:/bricks11/2 Brick490: gqas005:/bricks11/2 Brick491: gqas006:/bricks11/2 Brick492: gqas008:/bricks11/2 Brick493: gqas013:/bricks11/3 Brick494: gqas014:/bricks11/3 Brick495: gqas015:/bricks11/3 Brick496: gqas005:/bricks11/3 Brick497: gqas006:/bricks11/3 Brick498: gqas008:/bricks11/3 Brick499: gqas013:/bricks11/4 Brick500: gqas014:/bricks11/4 Brick501: gqas015:/bricks11/4 Brick502: gqas005:/bricks11/4 Brick503: gqas006:/bricks11/4 Brick504: gqas008:/bricks11/4 Brick505: gqas013:/bricks11/5 Brick506: gqas014:/bricks11/5 Brick507: gqas015:/bricks11/5 Brick508: gqas005:/bricks11/5 Brick509: gqas006:/bricks11/5 Brick510: gqas008:/bricks11/5 Brick511: gqas013:/bricks11/6 Brick512: gqas014:/bricks11/6 Brick513: gqas015:/bricks11/6 Brick514: gqas005:/bricks11/6 Brick515: gqas006:/bricks11/6 Brick516: gqas008:/bricks11/6 Brick517: gqas013:/bricks11/7 Brick518: gqas014:/bricks11/7 Brick519: gqas015:/bricks11/7 Brick520: gqas005:/bricks11/7 Brick521: gqas006:/bricks11/7 Brick522: gqas008:/bricks11/7 Brick523: gqas013:/bricks11/8 Brick524: gqas014:/bricks11/8 Brick525: gqas015:/bricks11/8 Brick526: gqas005:/bricks11/8 Brick527: gqas006:/bricks11/8 Brick528: gqas008:/bricks11/8 Brick529: gqas013:/bricks12/1 Brick530: gqas014:/bricks12/1 Brick531: gqas015:/bricks12/1 Brick532: gqas005:/bricks12/1 Brick533: gqas006:/bricks12/1 Brick534: gqas008:/bricks12/1 Brick535: gqas013:/bricks12/2 Brick536: gqas014:/bricks12/2 Brick537: gqas015:/bricks12/2 Brick538: gqas005:/bricks12/2 Brick539: gqas006:/bricks12/2 Brick540: gqas008:/bricks12/2 Brick541: gqas013:/bricks12/3 Brick542: gqas014:/bricks12/3 Brick543: gqas015:/bricks12/3 Brick544: gqas005:/bricks12/3 Brick545: gqas006:/bricks12/3 Brick546: gqas008:/bricks12/3 Brick547: gqas013:/bricks12/4 Brick548: gqas014:/bricks12/4 Brick549: gqas015:/bricks12/4 Brick550: gqas005:/bricks12/4 Brick551: gqas006:/bricks12/4 Brick552: gqas008:/bricks12/4 Brick553: gqas013:/bricks12/5 Brick554: gqas014:/bricks12/5 Brick555: gqas015:/bricks12/5 Brick556: gqas005:/bricks12/5 Brick557: gqas006:/bricks12/5 Brick558: gqas008:/bricks12/5 Brick559: gqas013:/bricks12/6 Brick560: gqas014:/bricks12/6 Brick561: gqas015:/bricks12/6 Brick562: gqas005:/bricks12/6 Brick563: gqas006:/bricks12/6 Brick564: gqas008:/bricks12/6 Brick565: gqas013:/bricks12/7 Brick566: gqas014:/bricks12/7 Brick567: gqas015:/bricks12/7 Brick568: gqas005:/bricks12/7 Brick569: gqas006:/bricks12/7 Brick570: gqas008:/bricks12/7 Brick571: gqas013:/bricks12/8 Brick572: gqas014:/bricks12/8 Brick573: gqas015:/bricks12/8 Brick574: gqas005:/bricks12/8 Brick575: gqas006:/bricks12/8 Brick576: gqas008:/bricks12/8 Options Reconfigured: performance.parallel-readdir: disable network.inode-lru-limit: 50000 performance.md-cache-timeout: 600 performance.cache-invalidation: on performance.stat-prefetch: on features.cache-invalidation-timeout: 600 features.cache-invalidation: on transport.address-family: inet nfs.disable: on [root@gqas013 ~]#
rms and finds are a part of regular testing. I am positive that this passed on 3.2. Marking as Regression.
*BT with missing debug infos installed * : (gdb) bt #0 frame_fill_groups (frame=frame@entry=0x7fa864080ac0) at fuse-helpers.c:158 #1 0x00007fa89fcea1d6 in get_groups (frame=0x7fa864080ac0, priv=0x7fa8a96e2040) at fuse-helpers.c:321 #2 get_call_frame_for_req (state=state@entry=0x7fa87c0065e0) at fuse-helpers.c:366 #3 0x00007fa89fcf27d0 in fuse_unlink_resume (state=0x7fa87c0065e0) at fuse-bridge.c:1631 #4 0x00007fa89fcec5c5 in fuse_resolve_done (state=<optimized out>) at fuse-resolve.c:663 #5 fuse_resolve_all (state=<optimized out>) at fuse-resolve.c:690 #6 0x00007fa89fcec2d8 in fuse_resolve (state=0x7fa87c0065e0) at fuse-resolve.c:654 #7 0x00007fa89fcec60e in fuse_resolve_all (state=<optimized out>) at fuse-resolve.c:686 #8 0x00007fa89fceb8f3 in fuse_resolve_continue (state=state@entry=0x7fa87c0065e0) at fuse-resolve.c:706 #9 0x00007fa89fcebae7 in fuse_resolve_entry_cbk (frame=<optimized out>, cookie=<optimized out>, this=0x7fa8a96dbef0, op_ret=0, op_errno=0, inode=0x7fa8880465f0, buf=0x7fa892ffcc60, xattr=0x0, postparent=0x7fa892ffccd0) at fuse-resolve.c:76 #10 0x00007fa899645069 in io_stats_lookup_cbk (frame=0x7fa87d40e630, cookie=<optimized out>, this=<optimized out>, op_ret=0, op_errno=0, inode=0x7fa8880465f0, buf=0x7fa892ffcc60, xdata=0x0, postparent=0x7fa892ffccd0) at io-stats.c:2190 #11 0x00007fa8a890c4d1 in default_lookup_cbk (frame=frame@entry=0x7fa87c055060, cookie=<optimized out>, this=<optimized out>, op_ret=op_ret@entry=0, op_errno=op_errno@entry=0, inode=0x7fa8880465f0, buf=buf@entry=0x7fa892ffcc60, xdata=0x0, postparent=postparent@entry=0x7fa892ffccd0) at defaults.c:1265 #12 0x00007fa899a70933 in mdc_lookup (frame=0x7fa864080ac0, this=<optimized out>, loc=0x7fa87c17cfa0, xdata=<optimized out>) at md-cache.c:1123 #13 0x00007fa8a8920b92 in default_lookup_resume (frame=0x7fa87c055060, this=0x7fa89401d280, loc=0x7fa87c17cfa0, xdata=0x0) at defaults.c:1872 #14 0x00007fa8a88b0b25 in call_resume (stub=0x7fa87c17cf50) at call-stub.c:2508 #15 0x00007fa89985b957 in iot_worker (data=0x7fa89402c900) at io-threads.c:220 #16 0x00007fa8a76eddc5 in start_thread (arg=0x7fa892ffd700) at pthread_create.c:308 #17 0x00007fa8a703273d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 (gdb)
(In reply to Ambarish from comment #0) > Description of problem: > ------------------------ > > The problem is fairly reproducible on EC 96*(4+2) and on a small 2*2 volume > as well,with millions and millions of files. > > Ran find from a single mount. > > Karan could repro this on 2*2 as well during rm. crash during rm is filed under bz 1454416 and is fixed. However, Amabrish mentions that he hit this crash without any rm -rf operations involved. So, this is a different issue.
Steps to Repro : Create data set with lost of file on a FUSE mount: python /small-files/smallfile/smallfile_cli.py --operation create --threads 8 --file-size 64 --files 10000 --files-per-dir 100000 --top /gluster-mount --host-set <client list> Drop caches and run ll-R/find . -mindepth 1 -type f/rm -rf * on mount
(In reply to Ambarish from comment #7) > *BT with missing debug infos installed * : > > > (gdb) bt > #0 frame_fill_groups (frame=frame@entry=0x7fa864080ac0) at > fuse-helpers.c:158 > #1 0x00007fa89fcea1d6 in get_groups (frame=0x7fa864080ac0, > priv=0x7fa8a96e2040) at fuse-helpers.c:321 > #2 get_call_frame_for_req (state=state@entry=0x7fa87c0065e0) at > fuse-helpers.c:366 > #3 0x00007fa89fcf27d0 in fuse_unlink_resume (state=0x7fa87c0065e0) at > fuse-bridge.c:1631 > #4 0x00007fa89fcec5c5 in fuse_resolve_done (state=<optimized out>) at > fuse-resolve.c:663 > #5 fuse_resolve_all (state=<optimized out>) at fuse-resolve.c:690 > #6 0x00007fa89fcec2d8 in fuse_resolve (state=0x7fa87c0065e0) at > fuse-resolve.c:654 > #7 0x00007fa89fcec60e in fuse_resolve_all (state=<optimized out>) at > fuse-resolve.c:686 > #8 0x00007fa89fceb8f3 in fuse_resolve_continue > (state=state@entry=0x7fa87c0065e0) at fuse-resolve.c:706 > #9 0x00007fa89fcebae7 in fuse_resolve_entry_cbk (frame=<optimized out>, > cookie=<optimized out>, this=0x7fa8a96dbef0, op_ret=0, op_errno=0, > inode=0x7fa8880465f0, buf=0x7fa892ffcc60, xattr=0x0, > postparent=0x7fa892ffccd0) at fuse-resolve.c:76 > #10 0x00007fa899645069 in io_stats_lookup_cbk (frame=0x7fa87d40e630, > cookie=<optimized out>, this=<optimized out>, op_ret=0, op_errno=0, > inode=0x7fa8880465f0, buf=0x7fa892ffcc60, xdata=0x0, > postparent=0x7fa892ffccd0) at io-stats.c:2190 > #11 0x00007fa8a890c4d1 in default_lookup_cbk > (frame=frame@entry=0x7fa87c055060, cookie=<optimized out>, this=<optimized > out>, op_ret=op_ret@entry=0, op_errno=op_errno@entry=0, > inode=0x7fa8880465f0, > buf=buf@entry=0x7fa892ffcc60, xdata=0x0, > postparent=postparent@entry=0x7fa892ffccd0) at defaults.c:1265 > #12 0x00007fa899a70933 in mdc_lookup (frame=0x7fa864080ac0, this=<optimized > out>, loc=0x7fa87c17cfa0, xdata=<optimized out>) at md-cache.c:1123 > #13 0x00007fa8a8920b92 in default_lookup_resume (frame=0x7fa87c055060, > this=0x7fa89401d280, loc=0x7fa87c17cfa0, xdata=0x0) at defaults.c:1872 > #14 0x00007fa8a88b0b25 in call_resume (stub=0x7fa87c17cf50) at > call-stub.c:2508 > #15 0x00007fa89985b957 in iot_worker (data=0x7fa89402c900) at > io-threads.c:220 > #16 0x00007fa8a76eddc5 in start_thread (arg=0x7fa892ffd700) at > pthread_create.c:308 > #17 0x00007fa8a703273d in clone () at > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > (gdb) Listing the code at the point of crash: 158 char *saveptr = NULL; (gdb) l 153 char line[4096]; 154 char *ptr = NULL; 155 FILE *fp = NULL; 156 int idx = 0; 157 long int id = 0; 158 char *saveptr = NULL; 159 char *endptr = NULL; 160 int ret = 0; 161 int ngroups = FUSE_MAX_AUX_GROUPS; 162 gid_t mygroups[GF_MAX_AUX_GROUPS]; Ie. it's reported to happen in the declaration boilerplate of frame_fill_groups(). That is the same as can be observed in Bug 1464327, see the analysis there in Comment 2. We can conclude that it's the same stack overflow issue.
In Bug 1464327 we identified change I7ede90d0e41bcf55755cced5747fa0fb1699edb2 (https://review.gluster.org/#/q/I7ede90d0e41bcf55755cced5747fa0fb1699edb2) as the culprit. That change has been ported back to RHGS 3.1.2 so all RHGS version from 3.1.2 on are affected.
upstream patch : https://review.gluster.org/17706
downstream patch : https://code.engineering.redhat.com/gerrit/#/c/111305/
Verified on 3.8.4-35. Client process did not crash on multiple tries of single/multi-threaded rms/finds from various FUSE mounts.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:2774