(In reply to Chris King from comment #0)
> Description of problem:
> 
> after completing "remove-brick", the removed drive is still full of the
> hidden "dot files" (eg .htaccess). However, the dot files where still
> relocated to new bricks on the volume (not lost).
> 
> 
> Version-Release number of selected component (if applicable):
> 
> glusterfs 4.0.2
> 
> 
> How reproducible:
> 
> I have only run this once, but both bricks still contained the all of the
> hidden dot files.
> 
> 
> Steps to Reproduce:
> 1. ensure that you have hidden files on volume
> 2. gluster volume remove-brick conoak-photos
> liberator:/mnt/disk1/conoak-photos-data
> liberator:/mnt/disk3/conoak-photos-data start
> 3. gluster volume remove-brick conoak-photos
> liberator:/mnt/disk1/conoak-photos-data
> liberator:/mnt/disk3/conoak-photos-data status
> 4. wait until status is "completed"
> 5. gluster volume remove-brick conoak-photos
> liberator:/mnt/disk1/conoak-photos-data
> liberator:/mnt/disk3/conoak-photos-data commit
> 6. find /mnt/disk1 -type f | grep -v "/[.]glusterfs/"
> 7. find /mnt/disk3 -type f | grep -v "/[.]glusterfs/"
> 
> 
> Actual results:
> 
> hundreds of files still be found on the removed brick
> eg
> /mnt/disk1/conoak-photos-data/Photos/2003/2003-05-02/downsize/.picasa.ini
> /mnt/disk1/conoak-photos-data/Photos/2003/2003-05-02/.picasa.ini
> /mnt/disk1/conoak-photos-data/Photos/2003/2003-05-25/.picasa.ini
> /mnt/disk1/conoak-photos-data/Photos/2003/2003-05-30/.picasa.ini
> 
> 
> Expected results:
> 
> Expected no files on the removed brick.
> 
> 
> Additional info:
> 
> one of the bricks has zero sized files and had the "T" attribute, the other
> brick and volume where original size;
> 
> ls -la /mnt/disk1/conoak-photos-data/Photos/2003/2003-05-02/.picasa.ini
> ---------T 2 chris nasusers 0 Mar  7 07:10
> /mnt/disk1/conoak-photos-data/Photos/2003/2003-05-02/.picasa.ini
> 
> ls -la /mnt/disk5/conoak-photos-data/Photos/2003/2003-05-02/.picasa.ini
> -rw-rw-r-- 2 chris nasusers 323 Jan 24  2013
> /mnt/disk5/conoak-photos-data/Photos/2003/2003-05-02/.picasa.ini
> 
> ls -la /mnt/gluster-conoak-photos/Photos/2003/2003-05-02/.picasa.ini
> -rw-rw-r-- 1 chris nasusers 323 Jan 24  2013
> /mnt/gluster-conoak-photos/Photos/2003/2003-05-02/.picasa.ini


Are all the files on the removed bricks zero sized 'T' files? If yes, these are internal files and can be ignored.

> Are all the files on the removed bricks zero sized 'T' files? If yes, these
> are internal files and can be ignored.

just on one brick, the second brick still had the original files

(In reply to Chris King from comment #2)
> > Are all the files on the removed bricks zero sized 'T' files? If yes, these
> > are internal files and can be ignored.
> 
> just on one brick, the second brick still had the original files

Hi Chris,

I need more info to figure this out. IIUC,the bricks that were removed were disk1 and disk3. So disk5, where you see the original files, is still a part of your volume.

gluster volume remove-brick conoak-photos liberator:/mnt/disk1/conoak-photos-data liberator:/mnt/disk3/conoak-photos-data start


What is the output of gluster volume info <volname>?
Having zero sized 'T' files on the removed bricks (disk1 and disk3) is not a problem.

(In reply to Nithya Balachandran from comment #3)
> Hi Chris,
> 
> I need more info to figure this out. IIUC,the bricks that were removed were
> disk1 and disk3. So disk5, where you see the original files, is still a part
> of your volume.

Nithya, the overall exercise was to add two 8TB SAS drives, expand the volumes over bricks on these drives (which went well) and then remove two two 6TB SATA drives.

I should have shown the listing of disk3 (not disk 5) my mistake. Disk 3 was also zero sized.

# ls -la /mnt/disk3/conoak-photos-data/Photos/2003/2003-05-02/.picasa.ini
---------T 2 chris nasusers 0 Mar  7 07:10 /mnt/disk3/conoak-photos-data/Photos/2003/2003-05-02/.picasa.ini


> What is the output of gluster volume info <volname>?

# gluster volume info conoak-photos-data
Volume conoak-photos-data does not exist
[root@liberator ~]# gluster volume info conoak-photos

Volume Name: conoak-photos
Type: Distributed-Replicate
Volume ID: 9d5c2238-6f37-4f75-a2ea-5348b174fbb1
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: liberator:/mnt/disk5/conoak-photos-data
Brick2: liberator:/mnt/disk7/conoak-photos-data
Brick3: liberator:/mnt/disk2/conoak-photos-data
Brick4: liberator:/mnt/disk6/conoak-photos-data
Options Reconfigured:
changelog.changelog: on
geo-replication.ignore-pid-check: on
geo-replication.indexing: on
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off


> Having zero sized 'T' files on the removed bricks (disk1 and disk3) is not a
> problem.

OK its not a serious, but it does not seem right that all files except the "dot files" are removed completely and only the dot files remain (even if they are zero sized)

(In reply to Chris King from comment #4)
> (In reply to Nithya Balachandran from comment #3)
> > Hi Chris,
> > 
> > I need more info to figure this out. IIUC,the bricks that were removed were
> > disk1 and disk3. So disk5, where you see the original files, is still a part
> > of your volume.
> 
> Nithya, the overall exercise was to add two 8TB SAS drives, expand the
> volumes over bricks on these drives (which went well) and then remove two
> two 6TB SATA drives.
> 
> I should have shown the listing of disk3 (not disk 5) my mistake. Disk 3 was
> also zero sized.
> 
> # ls -la /mnt/disk3/conoak-photos-data/Photos/2003/2003-05-02/.picasa.ini
> ---------T 2 chris nasusers 0 Mar  7 07:10
> /mnt/disk3/conoak-photos-data/Photos/2003/2003-05-02/.picasa.ini
> 
> 
> > What is the output of gluster volume info <volname>?
> 
> # gluster volume info conoak-photos-data
> Volume conoak-photos-data does not exist
> [root@liberator ~]# gluster volume info conoak-photos
> 
> Volume Name: conoak-photos
> Type: Distributed-Replicate
> Volume ID: 9d5c2238-6f37-4f75-a2ea-5348b174fbb1
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 2 x 2 = 4
> Transport-type: tcp
> Bricks:
> Brick1: liberator:/mnt/disk5/conoak-photos-data
> Brick2: liberator:/mnt/disk7/conoak-photos-data
> Brick3: liberator:/mnt/disk2/conoak-photos-data
> Brick4: liberator:/mnt/disk6/conoak-photos-data
> Options Reconfigured:
> changelog.changelog: on
> geo-replication.ignore-pid-check: on
> geo-replication.indexing: on
> transport.address-family: inet
> nfs.disable: on
> performance.client-io-threads: off
> 
> 
> > Having zero sized 'T' files on the removed bricks (disk1 and disk3) is not a
> > problem.
> 
> OK its not a serious, but it does not seem right that all files except the
> "dot files" are removed completely and only the dot files remain (even if
> they are zero sized)

It is possible that those linkto files (the 'T' files) were already present on those bricks for some reason (maybe the files were renamed at some point).  Gluster will not migrate linkto files - there is no point doing so as they are internal files.


I'm closing this BZ as this is not a bug.