| Summary: | No space left on device error | ||
|---|---|---|---|
| Product: | [Community] GlusterFS | Reporter: | Anush Shetty <anush> |
| Component: | distribute | Assignee: | Amar Tumballi <amarts> |
| Status: | CLOSED WORKSFORME | QA Contact: | |
| Severity: | high | Docs Contact: | |
| Priority: | low | ||
| Version: | mainline | CC: | gluster-bugs, vijay, vraman |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | Type: | --- | |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
PATCH: http://patches.gluster.com/patch/4735 in master (No space left on device error) This is due to extended attributes getting filled up root@pitta:/mnt/export2# getfattr -d -m . . # file: . trusted.1004_14_09-posix.gen=0sTI77ZAAAAAE= trusted.1206_13_09-posix.gen=0sTI3GhgAAAAE= trusted.1208_13_09-posix.gen=0sTI3G6wAAAAE= trusted.1209_13_09-posix.gen=0sTI3HMgAAAAE= trusted.1215_13_09-posix.gen=0sTI3ImwAAAAE= trusted.1402_10_09-posix.gen=0sTIntOgAAAAE= trusted.1613_10_09-posix.gen=0sTIoL/QAAAAE= trusted.1650_13_09-posix.gen=0sTI4JOAAAAAE= trusted.1710_13_09-posix.gen=0sTI4NwAAAAAE= trusted.1714_09_09-posix.gen=0sTIjIowAAAAE= trusted.1720_09_09-posix.gen=0sTIjKCQAAAAE= trusted.1724_09_09-posix.gen=0sTIjK7AAAAAE= trusted.1749_13_09-posix.gen=0sTI4W3gAAAAE= trusted.1810_13_09-posix.gen=0sTI4bwQAAAAE= trusted.1815_13_09-posix.gen=0sTI4c4wAAAAE= trusted.1816_13_09-posix.gen=0sTI4dMQAAAAE= trusted.1817_13_09-posix.gen=0sTI4djQAAAAE= trusted.1822_09_09-posix.gen=0sTIjYkAAAAAE= trusted.1851_13_09-posix.gen=0sTI4lcgAAAAE= trusted.1926_09_09-posix.gen=0sTIjnvgAAAAE= trusted.afr.1111_14_09-client-0=0sAAAAAAAAAAAAAAAA trusted.afr.1111_14_09-client-1=0sAAAAAAAAAAAAAAAA trusted.afr.1111_14_09-locks=0sAAAAAAAAAAAAAAAA trusted.afr.1111_14_09-replace-brick=0sAAAAAAAAAAAAACcQ trusted.afr.1135_14_09-locks=0sAAAAAAAAAAEAAAAA trusted.afr.1135_14_09-replace-brick=0sAAAAAAAAAjkAAAR2 trusted.afr.1143_14_09-client-0=0sAAAAAAAAAAAAAAAA trusted.afr.1143_14_09-client-1=0sAAAAAAAAAAAAAAAA trusted.afr.1143_14_09-locks=0sAAAAAAAAAAAAAAAA trusted.afr.1143_14_09-replace-brick=0sAAAAAAAAAAAAAFcy trusted.afr.1154_14_09-client-0=0sAAAAAAAAAAAAAAAA trusted.afr.1154_14_09-client-1=0sAAAAAAAAAAAAAAAA trusted.afr.1154_14_09-locks=0sAAAAAAAAAAAAAAAA trusted.afr.1154_14_09-replace-brick=0sAAAAAAAAAAAAABPC trusted.afr.1159_14_09-client-0=0sAAAAAAAAAAAAAAAA trusted.afr.1159_14_09-client-1=0sAAAAAAAAAAAAAAAA trusted.afr.1159_14_09-locks=0sAAAAAAAAAAAAAAAA trusted.afr.1159_14_09-replace-brick=0sAAAAAAAAAAAAAE4g trusted.afr.1206_13_09-client-0=0sAAAAAAAAAAEAAAAA trusted.afr.1206_13_09-client-1=0sAAAAAAAAAAAAAAAA trusted.afr.1206_13_09-locks=0sAAAAAAAAAAAAAAAA trusted.afr.1206_13_09-replace-brick=0sAAAAAAAAAAEAAAAA trusted.afr.1206_14_09-client-0=0sAAAAAAAAAAAAAAAA trusted.afr.1206_14_09-client-1=0sAAAAAAAAAAAAAAAA trusted.afr.1208_13_09-client-0=0sAAAAAAAAAAAAAAAA trusted.afr.1208_13_09-client-1=0sAAAAAAAAAAAAAAAA trusted.afr.1208_13_09-locks=0sAAAAAAAAAAAAAAAA trusted.afr.1208_13_09-replace-brick=0sAAAAAAAAAAEAAAAA trusted.afr.1209_13_09-client-0=0sAAAAAAAAAAAAAAAA trusted.afr.1209_13_09-client-1=0sAAAAAAAAAAAAAAAA trusted.afr.1209_13_09-locks=0sAAAAAAAAAAAAAAAA trusted.afr.1209_13_09-replace-brick=0sAAAAAAAAAAAAACcQ trusted.afr.1215_13_09-client-0=0sAAAAAAAAAAAAAAAA trusted.afr.1215_13_09-client-1=0sAAAAAAAAAAAAAAAA trusted.afr.1215_13_09-locks=0sAAAAAAAAAAAAAAAA trusted.afr.1215_13_09-replace-brick=0sAAAAAAAAAAAAACcQ trusted.afr.1402_10_09-locks=0sAAAAAAAAAAAAAAAA trusted.afr.1402_10_09-replace-brick=0sAAAAAAAAAAEAAADo trusted.afr.1650_13_09-client-0=0sAAAAAAAAAAAAAAAA trusted.afr.1650_13_09-client-1=0sAAAAAAAAAAAAAAAA trusted.afr.1650_13_09-locks=0sAAAAAAAAAAAAAAAA trusted.afr.1650_13_09-replace-brick=0sAAAAAAAAAAAAAB3Y trusted.afr.1714_09_09-client-0=0sAAAAAAAAAAAAAAAA trusted.afr.1714_09_09-client-1=0sAAAAAAAAAAAAAAAA trusted.afr.1714_09_09-locks=0sAAAAAAAAAAAAAAAA trusted.afr.1714_09_09-replace-brick=0sAAAAAAAAAAAAACcQ trusted.afr.1724_09_09-client-0=0sAAAAAAAAAAAAAAAA trusted.afr.1724_09_09-client-1=0sAAAAAAAAAAAAAAAA trusted.afr.1724_09_09-locks=0sAAAAAAAAAAAAAAAA trusted.afr.1724_09_09-replace-brick=0sAAAAAAAAAAAAACcQ trusted.afr.1822_09_09-client-0=0sAAAAAAAAAAAAAAAA trusted.afr.1822_09_09-client-1=0sAAAAAAAAAAAAAAAA trusted.afr.1822_09_09-locks=0sAAAAAAAAAAAAAAAA trusted.afr.1822_09_09-replace-brick=0sAAAAAAAAAAAAAE4g trusted.afr.1851_13_09-locks=0sAAAAAAAAAAEAAAAA trusted.afr.1851_13_09-replace-brick=0sAAAAAAAAAWoAAAF7 trusted.afr.1926_09_09-client-0=0sAAAAAAAAAAAAAAAA trusted.afr.1926_09_09-client-1=0sAAAAAAAAAAAAAAAA trusted.afr.1926_09_09-locks=0sAAAAAAAAAAAAAAAA trusted.afr.1926_09_09-replace-brick=0sAAAAAAAAAAAAAAAK trusted.gfid=0sAAAAAAAAAAAAAAAAAAAAAQ== trusted.glusterfs.dht=0sAAAAAQAAAAAAAAAAf////g== trusted.glusterfs.test="working\000" root@pitta:/mnt/export2# getfattr -d -m . . | wc This issue won't come in case of having a fresh backend.. (or even not overused backend for different volumes). Hence will close this issue for now. Will open a bug about extended attribute cleanup script.. |
This was on a dht+afr setup. Even touching a file gives this error domU-12-31-39-14-04-C2:/mnt/gluster# touch dot touch: cannot touch `dot': No space left on device domU-12-31-39-14-04-C2:/mnt/gluster# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda1 9.9G 2.5G 6.9G 27% / tmpfs 3.8G 0 3.8G 0% /lib/init/rw udev 3.8G 24K 3.8G 1% /dev tmpfs 3.8G 0 3.8G 0% /dev/shm /dev/sdb 414G 17G 377G 5% /mnt glusterfs#domU-12-31-39-14-04-C2:1156_10_09 827G 33G 753G 5% /mnt/gluster [2010-09-10 11:57:43.572344] D [dht-common.c:232:dht_revalidate_cbk] 1156_10_09-dht: subvolume 1156_10_09-replicate-1 returned -1 (No space left on device) [2010-09-10 11:57:43.575005] D [dht-common.c:1889:dht_fd_cbk] 1156_10_09-dht: subvolume 1156_10_09-replicate-1 returned -1 (No space left on device) [2010-09-10 11:57:43.579051] D [dht-common.c:232:dht_revalidate_cbk] 1156_10_09-dht: subvolume 1156_10_09-replicate-1 returned -1 (No space left on device) [2010-09-10 11:57:43.579186] D [client3_1-fops.c:4745:client3_1_readdirp] 1156_10_09-client-0: (1): failed to get fd ctx. EBADFD [2010-09-10 11:57:43.579251] D [client3_1-fops.c:4745:client3_1_readdirp] 1156_10_09-client-3: (1): failed to get fd ctx. EBADFD [2010-09-10 11:57:43.579333] D [stat-prefetch.c:3581:sp_release] 1156_10_09: cache hits: 0, cache miss: 0 [2010-09-10 11:59:49.697502] W [dht-diskusage.c:216:dht_is_subvol_filled] 1156_10_09-dht: disk space on subvolume '1156_10_09-replicate-0' is getting full (100.00 %), consider adding more nodes [2010-09-10 11:59:49.697555] D [dht-diskusage.c:253:dht_free_disk_available_subvol] 1156_10_09-dht: no subvolume has enough free space to create [2010-09-10 11:59:49.697888] D [afr-transaction.c:960:afr_post_nonblocking_entrylk_cbk] 1156_10_09-replicate-0: Non blocking entrylks done. Proceeding to FOP [2010-09-10 11:59:49.699761] E [afr-transaction.c:699:afr_changelog_pre_op_cbk] 1156_10_09-replicate-0: xattrop failed on child 1156_10_09-client-0: No space left on device [2010-09-10 11:59:49.699936] E [afr-transaction.c:699:afr_changelog_pre_op_cbk] 1156_10_09-replicate-0: xattrop failed on child 1156_10_09-client-2: No space left on device [2010-09-10 11:59:49.699967] E [afr-transaction.c:699:afr_changelog_pre_op_cbk] 1156_10_09-replicate-0: xattrop failed on child 1156_10_09-client-1: No space left on device [2010-09-10 11:59:49.699987] D [afr-lk-common.c:410:transaction_lk_op] 1156_10_09-replicate-0: lk op is for a transaction [2010-09-10 11:59:49.700245] W [fuse-bridge.c:1500:fuse_create_cbk] glusterfs-fuse: 511: /dot => -1 (No space left on device) [2010-09-10 11:59:50.859936] D [dht-common.c:232:dht_revalidate_cbk] 1156_10_09-dht: subvolume 1156_10_09-replicate-1 returned -1 (No space left on device) [2010-09-10 11:59:50.863831] D [dht-common.c:1889:dht_fd_cbk] 1156_10_09-dht: subvolume 1156_10_09-replicate-1 returned -1 (No space left on device) [2010-09-10 11:59:50.869349] D [dht-common.c:232:dht_revalidate_cbk] 1156_10_09-dht: subvolume 1156_10_09-replicate-1 returned -1 (No space left on device) [2010-09-10 11:59:50.869552] D [client3_1-fops.c:4745:client3_1_readdirp] 1156_10_09-client-0: (1): failed to get fd ctx. EBADFD [2010-09-10 11:59:50.869594] D [client3_1-fops.c:4745:client3_1_readdirp] 1156_10_09-client-3: (1): failed to get fd ctx. EBADFD [2010-09-10 11:59:50.869696] D [stat-prefetch.c:3581:sp_release] 1156_10_09: cache hits: 0, cache miss: 0 [2010-09-10 12:00:08.446079] D [dht-common.c:232:dht_revalidate_cbk] 1156_10_09-dht: subvolume 1156_10_09-replicate-1 returned -1 (No space left on device) [2010-09-10 12:00:08.449855] D [dht-common.c:232:dht_revalidate_cbk] 1156_10_09-dht: subvolume 1156_10_09-replicate-1 returned -1 (No space left on device)