Bug 1240925
| Summary: | Data Tiering: file marked for demotion is getting demoted even if the file is being accessed( a heating operation) | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Nag Pavan Chilakam <nchilaka> |
| Component: | tier | Assignee: | Joseph Elwin Fernandes <josferna> |
| Status: | CLOSED DUPLICATE | QA Contact: | Nag Pavan Chilakam <nchilaka> |
| Severity: | urgent | Docs Contact: | |
| Priority: | urgent | ||
| Version: | rhgs-3.1 | CC: | asrivast, rcyriac, rhs-bugs, sankarshan, storage-qa-internal, vagarwal |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | glusterfs-3.7.1-9 | Doc Type: | Bug Fix |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2015-12-22 08:53:42 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1202842 | ||
|
Description
Nag Pavan Chilakam
2015-07-08 07:19:38 UTC
Raised an upstream bug for upstream tracking 1240926 sosreports at rhsqe-repo:/home/repo/sosreports/bug.1240925 Crash seen 0 [2015-07-15 19:30:00.791978] I [MSGID: 109038] [tier.c:572:tier_build_migration_qfile] 0-vol1-tier-dht: Failed to remove /var/run/gluster/demotequeryfile-20722 [2015-07-15 19:30:00.792157] E [MSGID: 109051] [tier.c:614:tier_migrate_files_using_qfile] 0-tier: Failed opening /var/run/gluster/demotequeryfile-20722 for migration [2015-07-15 19:30:00.792314] E [MSGID: 109037] [tier.c:898:tier_start] 0-vol1-tier-dht: Demotion failed! [2015-07-15 19:30:51.251535] I [MSGID: 109028] [dht-rebalance.c:3029:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 3717.00 secs [2015-07-15 19:30:51.251645] I [MSGID: 109028] [dht-rebalance.c:3033:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 1, failures: 1, skipped: 0 [2015-07-15 19:31:55.603263] I [MSGID: 109028] [dht-rebalance.c:3029:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 3781.00 secs [2015-07-15 19:32:00.547529] I [MSGID: 109028] [dht-rebalance.c:3029:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 3786.00 secs [2015-07-15 19:32:00.812843] I [MSGID: 109038] [tier.c:572:tier_build_migration_qfile] 0-vol1-tier-dht: Failed to remove /var/run/gluster/demotequeryfile-20722 [2015-07-15 19:32:00.813010] E [MSGID: 109051] [tier.c:614:tier_migrate_files_using_qfile] 0-tier: Failed opening /var/run/gluster/demotequeryfile-20722 for migration [2015-07-15 19:32:00.813204] E [MSGID: 109037] [tier.c:898:tier_start] 0-vol1-tier-dht: Demotion failed! The message "I [MSGID: 109028] [dht-rebalance.c:3033:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 1, failures: 1, skipped: 0" repeated 2 times between [2015-07-15 19:30:51.251645] and [2015-07-15 19:32:00.547674] pending frames: frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) patchset: git://git.gluster.com/glusterfs.git signal received: 11 time of crash: 2015-07-15 19:32:00 configuration details: argp 1 backtrace 1 dlfcn 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 3.7.1 /usr/lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xb6)[0x7f1ef2f408e6] /usr/lib64/libglusterfs.so.0(gf_print_trace+0x33f)[0x7f1ef2f604af] /lib64/libc.so.6(+0x36ede326a0)[0x7f1ef18df6a0] /lib64/libc.so.6(+0x36edf336ef)[0x7f1ef19e06ef] /usr/lib64/libgfdb.so.0(gf_sql_query_function+0xdf)[0x7f1ee40e1dff] /usr/lib64/libgfdb.so.0(gf_sqlite3_find_recently_changed_files+0xd5)[0x7f1ee40e5445] /usr/lib64/libgfdb.so.0(find_recently_changed_files+0x4f)[0x7f1ee40e0def] /usr/lib64/glusterfs/3.7.1/xlator/cluster/tier.so(+0x54305)[0x7f1ee5122305] /usr/lib64/libglusterfs.so.0(dict_foreach_match+0x74)[0x7f1ef2f38234] /usr/lib64/libglusterfs.so.0(dict_foreach+0x18)[0x7f1ef2f382e8] /usr/lib64/glusterfs/3.7.1/xlator/cluster/tier.so(+0x55f9e)[0x7f1ee5123f9e] /lib64/libpthread.so.0(+0x36ee207a51)[0x7f1ef202ba51] /lib64/libc.so.6(clone+0x6d)[0x7f1ef199596d] --------- ^C [root@nchilaka-tier-02 glusterfs]# [root@nchilaka-tier-02 glusterfs]# [root@nchilaka-tier-02 glusterfs]# [root@nchilaka-tier-02 glusterfs]# [root@nchilaka-tier-02 glusterfs]# [root@nchilaka-tier-02 glusterfs]# tail vol1-tier.log /usr/lib64/libgfdb.so.0(gf_sql_query_function+0xdf)[0x7f1ee40e1dff] /usr/lib64/libgfdb.so.0(gf_sqlite3_find_recently_changed_files+0xd5)[0x7f1ee40e5445] /usr/lib64/libgfdb.so.0(find_recently_changed_files+0x4f)[0x7f1ee40e0def] /usr/lib64/glusterfs/3.7.1/xlator/cluster/tier.so(+0x54305)[0x7f1ee5122305] /usr/lib64/libglusterfs.so.0(dict_foreach_match+0x74)[0x7f1ef2f38234] /usr/lib64/libglusterfs.so.0(dict_foreach+0x18)[0x7f1ef2f382e8] /usr/lib64/glusterfs/3.7.1/xlator/cluster/tier.so(+0x55f9e)[0x7f1ee5123f9e] /lib64/libpthread.so.0(+0x36ee207a51)[0x7f1ef202ba51] /lib64/libc.so.6(clone+0x6d)[0x7f1ef199596d] Moving to failed_qa as crash was observed and tier rebalance failed.
steps:
1)created a volume and set following options
performance.io-cache: off
performance.quick-read: off
2)atached tier and set following
features.ctr-enabled: on
3)Now mounted on nfs
4)created a zero byte file
5)got demoted after some time due to idleness
6)touched the file to heat it and also created another file and kept writing to this file as below
[root@nchilaka-nfs-client-6 vol1]#
[root@nchilaka-nfs-client-6 vol1]# ########file seems to have got demoted...checking ctim########
[root@nchilaka-nfs-client-6 vol1]# stat hf1
File: `hf1'
Size: 0 Blocks: 0 IO Block: 1048576 regular empty file
Device: 15h/21d Inode: 13574446887746474160 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2015-07-16 00:01:19.000000000 +0530
Modify: 2015-07-16 00:01:19.000000000 +0530
Change: 2015-07-16 00:04:00.545000003 +0530
[root@nchilaka-nfs-client-6 vol1]#
[root@nchilaka-nfs-client-6 vol1]#
[root@nchilaka-nfs-client-6 vol1]# ###############now touching hf1########## and also creating a new file writing to it continurouslyu############
[root@nchilaka-nfs-client-6 vol1]# touch hf1
[root@nchilaka-nfs-client-6 vol1]# touch hf1
[root@nchilaka-nfs-client-6 vol1]# touch hf1
[root@nchilaka-nfs-client-6 vol1]# stat hf1
File: `hf1'
Size: 0 Blocks: 0 IO Block: 1048576 regular empty file
Device: 15h/21d Inode: 13574446887746474160 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2015-07-16 00:06:15.000000000 +0530
Modify: 2015-07-16 00:06:15.000000000 +0530
Change: 2015-07-16 00:06:15.668999992 +0530
[root@nchilaka-nfs-client-6 vol1]# touch hf2
[root@nchilaka-nfs-client-6 vol1]# for i in {1..300};do echo "$i:hello">> hf2;sleep 2;done
^C
[root@nchilaka-nfs-client-6 vol1]# sleep 70;for i in {1..300};do echo "$i:newnenwwnw .. hello">> hf2;sleep 115;done
^C
[root@nchilaka-nfs-client-6 vol1]#
Saw the crash.
Tested on the build version glusterfs-3.7.1-9
Logs are attached
Could you please share the sos reports I need to look into the crash as a crash could have happened because of other issues also. Please share the complete sos report sos reports @ rhsqe-repo.lab.eng.blr.redhat.com:/home/repo/sosreports/bug.1240925/failed_qa *** This bug has been marked as a duplicate of bug 1273347 *** |