Description of problem: ======================= I have a volume, where i had created some 5 zerobyte files and about 3 non-empty files. I later attached a tier, which means all the files reside in the cold tier, by default. After enabling ctr, I performed a read of all files one after one(and even tried parallelly issuing cat *). I observed that only about two zerobyte files were getting promoted and not all the files. This problem was seen consistently. Version-Release number of selected component (if applicable): ============================================================ [root@tettnang glusterfs]# gluster --version glusterfs 3.7.1 built on Jun 28 2015 11:01:14 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com> GlusterFS comes with ABSOLUTELY NO WARRANTY. You may redistribute copies of GlusterFS under the terms of the GNU General Public License. [root@tettnang glusterfs]# rpm -qa|grep gluster glusterfs-cli-3.7.1-6.el7rhgs.x86_64 glusterfs-client-xlators-3.7.1-6.el7rhgs.x86_64 glusterfs-fuse-3.7.1-6.el7rhgs.x86_64 glusterfs-geo-replication-3.7.1-6.el7rhgs.x86_64 glusterfs-libs-3.7.1-6.el7rhgs.x86_64 glusterfs-api-3.7.1-6.el7rhgs.x86_64 glusterfs-rdma-3.7.1-6.el7rhgs.x86_64 glusterfs-3.7.1-6.el7rhgs.x86_64 glusterfs-debuginfo-3.7.1-6.el7rhgs.x86_64 samba-vfs-glusterfs-4.1.17-7.el7rhgs.x86_64 glusterfs-server-3.7.1-6.el7rhgs.x86_64 Steps to Reproduce: =================== 1.create a dist-rep volume 2.start the volume and turn off following performance.io-cache: off performance.quick-read: off 3. mount volume and create some empty and non-empty files 4. Attach a distributed hot tier 5. Turn on features.ctr-enabled 6. do a ls -l(this will create T files of all cold files in hot tier for which bug#1238549 was raised) 7. Now read file contents of all files using cat, one after another without any time gap. Also you can do a parallel read by issing cat * Actual results: ============== It can be seen that only a couple of files are getting promoted(and in my case the files with data never got promoted) Expected results: =================== All files should get promoted, if accessed once atleast Sos and cli logs are attached
Created attachment 1045750 [details] cli logs while execution
CLi logs are attached. Sos reports available at rhsqe-repo.lab.eng.blr.redhat.com:/home/repo/sosreports/bug.1238944
I see the files getting selected for promote/demote, but due to permission denied on set xattr, the files are not getting migrated [2015-07-07 12:58:00.689838] I [MSGID: 109038] [tier.c:350:tier_migrate_using_query_file] 0-vol2-tier-dht: Tier 1 src_subvol vol2-cold-dht file file1 [2015-07-07 12:58:00.691438] I [dht-rebalance.c:1002:dht_migrate_file] 0-vol2-tier-dht: /file1: attempting to move from vol2-cold-dht to vol2-hot-dht [2015-07-07 12:58:00.704294] W [MSGID: 114031] [client-rpc-fops.c:1977:client3_3_fsetxattr_cbk] 0-vol2-client-5: remote operation failed [Permission denied] [2015-07-07 12:58:00.704453] W [MSGID: 109023] [dht-rebalance.c:546:__dht_rebalance_create_dst_file] 0-vol2-tier-dht: /file1: failed to set xattr on vol2-hot-dht (Permission denied) [2015-07-07 12:58:00.804434] I [MSGID: 109022] [dht-rebalance.c:1282:dht_migrate_file] 0-vol2-tier-dht: completed migration of /file1 from subvolume vol2-cold-dht to vol2-hot-dht
sos reports for above logs can be viewed at rhsqe-repo.lab.eng.blr.redhat.com:/home/repo/sosreports/bug.1240569
https://code.engineering.redhat.com/gerrit/#/c/60418/
Could the fixed_in_version field be updated, please?
fixed in 3.1.2