Cause:
When nameless lookup on a gfid came, posix was trying to resolve it using the gfid handle by doing readlink on it. The return value of readlink was received in a wrong variable.
Consequence:
The above cause made posix think readlink was successfu, despite the failure and continue leading to glusterfsd process being crashed.
Fix:
Now the return value of readlink call is received in a proper variable and checked for failure before continuing.
Result:
DescriptionRaghavendra Bhat
2013-08-27 12:33:28 UTC
Description of problem:
if the quota is set on a non existing directory, then glusterfsd process crashes with the below backtrace.
Simply trying to set quota limit on a non existing directory does not crash the brick process. To reproduce it, one has to add a new brick, perform rebalance, remove a directory upon which quota limit was set. Now trying to set limit on the removed directory will crash the glusterfsd process.
this is the backtrace
155 strcat (dir_name, "/");
Missing separate debuginfos, use: debuginfo-install glibc-2.15-59.fc17.x86_64 keyutils-libs-1.5.5-2.fc17.x86_64 krb5-libs-1.10.2-12.fc17.x86_64 libaio-0.3.109-5.fc17.x86_64 libcom_err-1.42.3-3.fc17.x86_64 libgcc-4.7.2-2.fc17.x86_64 libselinux-2.1.10-3.fc17.x86_64 openssl-1.0.0k-1.fc17.x86_64 zlib-1.2.5-7.fc17.x86_64
(gdb) bt
#0 0x00007fc7319ebff0 in posix_make_ancestryfromgfid (this=0x13bdef0, path=0x7fc7292117c0 "", pathsize=4097, head=0x7fc6fc002970, type=1,
gfid=0x7fc72a26d0d0 "\334\303\062\017\347\227G\205\253;\257\204\262\063$4\001", handle_size=67, priv_base_path=0x13eb720 "/export1/vol", itable=0x13f6970,
parent=0x7fc7292117b8, xdata=0x7fc73423624c) at ../../../../../xlators/storage/posix/src/posix-handle.c:155
#1 0x00007fc7319ddb4f in posix_get_ancestry_directory (this=0x13bdef0, real_path=0x7fc7292129e0 "/export1/vol/dir/", loc=0x7fc7342f24c8, dict=0x7fc734236cb0, type=1,
op_errno=0x7fc729212998, xdata=0x7fc73423624c) at ../../../../../xlators/storage/posix/src/posix.c:2764
#2 0x00007fc7319dea54 in posix_get_ancestry (this=0x13bdef0, real_path=0x7fc7292129e0 "/export1/vol/dir/", loc=0x7fc7342f24c8, dict=0x7fc734236cb0, type=1,
op_errno=0x7fc729212998, xdata=0x7fc73423624c) at ../../../../../xlators/storage/posix/src/posix.c:3084
#3 0x00007fc7319e8f89 in _posix_xattr_get_set (xattr_req=0x7fc73423624c, key=0x146da20 "glusterfs.ancestry.path", data=0x7fc734066b1c, xattrargs=0x7fc729212960)
at ../../../../../xlators/storage/posix/src/posix-helpers.c:319
#4 0x00007fc735a2fa87 in dict_foreach (dict=0x7fc73423624c, fn=0x7fc7319e8b5b <_posix_xattr_get_set>, data=0x7fc729212960) at ../../../libglusterfs/src/dict.c:1109
#5 0x00007fc7319e9a72 in posix_lookup_xattr_fill (this=0x13bdef0, real_path=0x7fc7292129e0 "/export1/vol/dir/", loc=0x7fc7342f24c8, xattr_req=0x7fc73423624c,
buf=0x7fc729212ac0) at ../../../../../xlators/storage/posix/src/posix-helpers.c:558
#6 0x00007fc7319cee8c in posix_lookup (frame=0x7fc734879824, this=0x13bdef0, loc=0x7fc7342f24c8, xdata=0x7fc73423624c)
at ../../../../../xlators/storage/posix/src/posix.c:156
#7 0x00007fc735a43190 in default_lookup (frame=0x7fc734879824, this=0x13bf720, loc=0x7fc7342f24c8, xdata=0x7fc73423624c) at ../../../libglusterfs/src/defaults.c:1253
#8 0x00007fc7315a8d79 in posix_acl_lookup (frame=0x7fc734883200, this=0x13c0910, loc=0x7fc7342f24c8, xattr=0x7fc73423624c)
at ../../../../../xlators/system/posix-acl/src/posix-acl.c:793
#9 0x00007fc731395445 in pl_lookup (frame=0x7fc734876dd0, this=0x13c19c0, loc=0x7fc7342f24c8, xdata=0x7fc73423624c)
at ../../../../../xlators/features/locks/src/posix.c:2081
#10 0x00007fc73116f5e1 in iot_lookup_wrapper (frame=0x7fc734882134, this=0x13c2a80, loc=0x7fc7342f24c8, xdata=0x7fc73423624c)
at ../../../../../xlators/performance/io-threads/src/io-threads.c:346
#11 0x00007fc735a58251 in call_resume_wind (stub=0x7fc7342f2488) at ../../../libglusterfs/src/call-stub.c:2312
#12 0x00007fc735a5f0a8 in call_resume (stub=0x7fc7342f2488) at ../../../libglusterfs/src/call-stub.c:2645
#13 0x00007fc73116efbe in iot_worker (data=0x13e7190) at ../../../../../xlators/performance/io-threads/src/io-threads.c:191
#14 0x0000003fc5807d14 in start_thread () from /lib64/libpthread.so.0
#15 0x0000003fc54f168d in clone () from /lib64/libc.so.6
(gdb) f 0
#0 0x00007fc7319ebff0 in posix_make_ancestryfromgfid (this=0x13bdef0, path=0x7fc7292117c0 "", pathsize=4097, head=0x7fc6fc002970, type=1,
gfid=0x7fc72a26d0d0 "\334\303\062\017\347\227G\205\253;\257\204\262\063$4\001", handle_size=67, priv_base_path=0x13eb720 "/export1/vol", itable=0x13f6970,
parent=0x7fc7292117b8, xdata=0x7fc73423624c) at ../../../../../xlators/storage/posix/src/posix-handle.c:155
155 strcat (dir_name, "/");
(gdb) p dir_name
$1 = 0x0
(gdb) l
150 len = readlink (dir_handle, linkname, 512);
151 linkname[len] = '\0';
152
153 pgfidstr = strtok_r (linkname + SLEN("../../00/00/"), "/", &saveptr);
154 dir_name = strtok_r (NULL, "/", &saveptr);
155 strcat (dir_name, "/");
156 uuid_parse (pgfidstr, tmp_gfid);
157
158 ret = posix_make_ancestryfromgfid (this, path, pathsize, head, type,
159 tmp_gfid, handle_size,
(gdb) p len
$2 = 18446744073709551615
(gdb) p dir_handle
$3 = 0x7fc7292115e0 "/export1/vol/.glusterfs/dc/c3/dcc3320f-e797-4785-ab3b-af84b2332434"
(gdb) p pgfidstr
$4 = 0x0
(gdb) l posix_make_ancestryfromgfid
105 posix_make_ancestryfromgfid (xlator_t *this, char *path, int pathsize,
106 gf_dirent_t *head, int type, uuid_t gfid,
107 const size_t handle_size,
108 const char *priv_base_path, inode_table_t *itable,
109 inode_t **parent, dict_t *xdata)
110 {
111 char *linkname = NULL; /* "../../<gfid[0]>/<gfid[1]/"
112 "<gfidstr>/<NAME_MAX>" */
113 char *dir_handle = NULL;
114 char *dir_name = NULL;
(gdb)
115 char *pgfidstr = NULL;
116 char *saveptr = NULL;
117 size_t len = 0;
118 inode_t *inode = NULL;
119 struct iatt iabuf = {0, };
120 int ret = -1;
121 uuid_t tmp_gfid = {0};
122
Above, len is declared as an unsigned entity. But readlink returns a -ve value upon error and len will become a huge +ve value if readlink fails. Also the return value of readlink is never checked.
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1.
2.
3.
Actual results:
Expected results:
Additional info:
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
http://rhn.redhat.com/errata/RHBA-2013-1769.html
Description of problem: if the quota is set on a non existing directory, then glusterfsd process crashes with the below backtrace. Simply trying to set quota limit on a non existing directory does not crash the brick process. To reproduce it, one has to add a new brick, perform rebalance, remove a directory upon which quota limit was set. Now trying to set limit on the removed directory will crash the glusterfsd process. this is the backtrace 155 strcat (dir_name, "/"); Missing separate debuginfos, use: debuginfo-install glibc-2.15-59.fc17.x86_64 keyutils-libs-1.5.5-2.fc17.x86_64 krb5-libs-1.10.2-12.fc17.x86_64 libaio-0.3.109-5.fc17.x86_64 libcom_err-1.42.3-3.fc17.x86_64 libgcc-4.7.2-2.fc17.x86_64 libselinux-2.1.10-3.fc17.x86_64 openssl-1.0.0k-1.fc17.x86_64 zlib-1.2.5-7.fc17.x86_64 (gdb) bt #0 0x00007fc7319ebff0 in posix_make_ancestryfromgfid (this=0x13bdef0, path=0x7fc7292117c0 "", pathsize=4097, head=0x7fc6fc002970, type=1, gfid=0x7fc72a26d0d0 "\334\303\062\017\347\227G\205\253;\257\204\262\063$4\001", handle_size=67, priv_base_path=0x13eb720 "/export1/vol", itable=0x13f6970, parent=0x7fc7292117b8, xdata=0x7fc73423624c) at ../../../../../xlators/storage/posix/src/posix-handle.c:155 #1 0x00007fc7319ddb4f in posix_get_ancestry_directory (this=0x13bdef0, real_path=0x7fc7292129e0 "/export1/vol/dir/", loc=0x7fc7342f24c8, dict=0x7fc734236cb0, type=1, op_errno=0x7fc729212998, xdata=0x7fc73423624c) at ../../../../../xlators/storage/posix/src/posix.c:2764 #2 0x00007fc7319dea54 in posix_get_ancestry (this=0x13bdef0, real_path=0x7fc7292129e0 "/export1/vol/dir/", loc=0x7fc7342f24c8, dict=0x7fc734236cb0, type=1, op_errno=0x7fc729212998, xdata=0x7fc73423624c) at ../../../../../xlators/storage/posix/src/posix.c:3084 #3 0x00007fc7319e8f89 in _posix_xattr_get_set (xattr_req=0x7fc73423624c, key=0x146da20 "glusterfs.ancestry.path", data=0x7fc734066b1c, xattrargs=0x7fc729212960) at ../../../../../xlators/storage/posix/src/posix-helpers.c:319 #4 0x00007fc735a2fa87 in dict_foreach (dict=0x7fc73423624c, fn=0x7fc7319e8b5b <_posix_xattr_get_set>, data=0x7fc729212960) at ../../../libglusterfs/src/dict.c:1109 #5 0x00007fc7319e9a72 in posix_lookup_xattr_fill (this=0x13bdef0, real_path=0x7fc7292129e0 "/export1/vol/dir/", loc=0x7fc7342f24c8, xattr_req=0x7fc73423624c, buf=0x7fc729212ac0) at ../../../../../xlators/storage/posix/src/posix-helpers.c:558 #6 0x00007fc7319cee8c in posix_lookup (frame=0x7fc734879824, this=0x13bdef0, loc=0x7fc7342f24c8, xdata=0x7fc73423624c) at ../../../../../xlators/storage/posix/src/posix.c:156 #7 0x00007fc735a43190 in default_lookup (frame=0x7fc734879824, this=0x13bf720, loc=0x7fc7342f24c8, xdata=0x7fc73423624c) at ../../../libglusterfs/src/defaults.c:1253 #8 0x00007fc7315a8d79 in posix_acl_lookup (frame=0x7fc734883200, this=0x13c0910, loc=0x7fc7342f24c8, xattr=0x7fc73423624c) at ../../../../../xlators/system/posix-acl/src/posix-acl.c:793 #9 0x00007fc731395445 in pl_lookup (frame=0x7fc734876dd0, this=0x13c19c0, loc=0x7fc7342f24c8, xdata=0x7fc73423624c) at ../../../../../xlators/features/locks/src/posix.c:2081 #10 0x00007fc73116f5e1 in iot_lookup_wrapper (frame=0x7fc734882134, this=0x13c2a80, loc=0x7fc7342f24c8, xdata=0x7fc73423624c) at ../../../../../xlators/performance/io-threads/src/io-threads.c:346 #11 0x00007fc735a58251 in call_resume_wind (stub=0x7fc7342f2488) at ../../../libglusterfs/src/call-stub.c:2312 #12 0x00007fc735a5f0a8 in call_resume (stub=0x7fc7342f2488) at ../../../libglusterfs/src/call-stub.c:2645 #13 0x00007fc73116efbe in iot_worker (data=0x13e7190) at ../../../../../xlators/performance/io-threads/src/io-threads.c:191 #14 0x0000003fc5807d14 in start_thread () from /lib64/libpthread.so.0 #15 0x0000003fc54f168d in clone () from /lib64/libc.so.6 (gdb) f 0 #0 0x00007fc7319ebff0 in posix_make_ancestryfromgfid (this=0x13bdef0, path=0x7fc7292117c0 "", pathsize=4097, head=0x7fc6fc002970, type=1, gfid=0x7fc72a26d0d0 "\334\303\062\017\347\227G\205\253;\257\204\262\063$4\001", handle_size=67, priv_base_path=0x13eb720 "/export1/vol", itable=0x13f6970, parent=0x7fc7292117b8, xdata=0x7fc73423624c) at ../../../../../xlators/storage/posix/src/posix-handle.c:155 155 strcat (dir_name, "/"); (gdb) p dir_name $1 = 0x0 (gdb) l 150 len = readlink (dir_handle, linkname, 512); 151 linkname[len] = '\0'; 152 153 pgfidstr = strtok_r (linkname + SLEN("../../00/00/"), "/", &saveptr); 154 dir_name = strtok_r (NULL, "/", &saveptr); 155 strcat (dir_name, "/"); 156 uuid_parse (pgfidstr, tmp_gfid); 157 158 ret = posix_make_ancestryfromgfid (this, path, pathsize, head, type, 159 tmp_gfid, handle_size, (gdb) p len $2 = 18446744073709551615 (gdb) p dir_handle $3 = 0x7fc7292115e0 "/export1/vol/.glusterfs/dc/c3/dcc3320f-e797-4785-ab3b-af84b2332434" (gdb) p pgfidstr $4 = 0x0 (gdb) l posix_make_ancestryfromgfid 105 posix_make_ancestryfromgfid (xlator_t *this, char *path, int pathsize, 106 gf_dirent_t *head, int type, uuid_t gfid, 107 const size_t handle_size, 108 const char *priv_base_path, inode_table_t *itable, 109 inode_t **parent, dict_t *xdata) 110 { 111 char *linkname = NULL; /* "../../<gfid[0]>/<gfid[1]/" 112 "<gfidstr>/<NAME_MAX>" */ 113 char *dir_handle = NULL; 114 char *dir_name = NULL; (gdb) 115 char *pgfidstr = NULL; 116 char *saveptr = NULL; 117 size_t len = 0; 118 inode_t *inode = NULL; 119 struct iatt iabuf = {0, }; 120 int ret = -1; 121 uuid_t tmp_gfid = {0}; 122 Above, len is declared as an unsigned entity. But readlink returns a -ve value upon error and len will become a huge +ve value if readlink fails. Also the return value of readlink is never checked. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: