+++ This bug was initially created as a clone of Bug #1224033 +++

Description of problem:


Version-Release number of selected component (if applicable):


How reproducible: 1:1


Steps to Reproduce:
1. Have a 2*2 distribute replicate volume 'vol1' on a 2node cluster
2. Create a glusterfind session 'sess' for 'vol1'
3. Mount 'vol1' over fuse, and copy a couple of files to the moutpoint
4. Run the glusterfind pre command for 'sess' and 'vol1'.


Expected: Step4 should get executed successfully, with /root/out.txt having logged the details of all files newly created, or modified.

Actual: Crash is observed at step4.


Additional info:

[root@dhcp43-140 ~]# 
[root@dhcp43-140 ~]# glusterfind pre sess vol1 out.txt
Post command is not run after last pre, use --regenerate-outfile
[root@dhcp43-140 ~]# 
[root@dhcp43-140 ~]# 
[root@dhcp43-140 ~]# glusterfind post sess vol1
[root@dhcp43-140 ~]# 
[root@dhcp43-140 ~]# vi /var/lib/glusterd/glusterfind/sess/vol1/
%2Frhs%2Fbrick1%2Fdd.status  %2Frhs%2Fbrick2%2Fdd.status  sess_vol1_secret.pem         sess_vol1_secret.pem.pub     status                       
[root@dhcp43-140 ~]# cat /var/lib/glusterd/glusterfind/sess/vol1/status
1432029619[root@dhcp43-140 ~]# 
[root@dhcp43-140 ~]# 
[root@dhcp43-140 ~]# glusterfind pre
usage: glusterfind pre [-h] [--debug] [--full] [--disable-partial]
                       [--output-prefix OUTPUT_PREFIX] [--regenerate-outfile]
                       [-N]
                       session volume outfile
glusterfind pre: error: too few arguments
[root@dhcp43-140 ~]# glusterfind pre sess vol1 out.txt
10.70.43.140 - pre failed: Traceback (most recent call last):
  File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 389, in <module>
    actual_end = changelog_crawl(args.brick, start, end, args)
  File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 340, in changelog_crawl
    return get_changes(brick, working_dir, log_file, start, end, args)
  File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 303, in get_changes
    pgfid_to_path(brick, changelog_data)
  File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 64, in pgfid_to_path
    path = symlink_gfid_to_path(brick, row[0])
  File "/usr/libexec/glusterfs/glusterfind/utils.py", line 184, in symlink_gfid_to_path
    path_readlink = os.readlink(path)
OSError: [Errno 2] No such file or directory: '/rhs/brick1/dd/.glusterfs/01/a9/01a90b40-713d-479f-8ecf-0373734a048f'

10.70.43.140 - pre failed: Traceback (most recent call last):
  File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 389, in <module>
    actual_end = changelog_crawl(args.brick, start, end, args)
  File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 340, in changelog_crawl
    return get_changes(brick, working_dir, log_file, start, end, args)
  File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 303, in get_changes
    pgfid_to_path(brick, changelog_data)
  File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 64, in pgfid_to_path
    path = symlink_gfid_to_path(brick, row[0])
  File "/usr/libexec/glusterfs/glusterfind/utils.py", line 184, in symlink_gfid_to_path
    path_readlink = os.readlink(path)
OSError: [Errno 2] No such file or directory: '/rhs/brick2/dd/.glusterfs/01/a9/01a90b40-713d-479f-8ecf-0373734a048f'

10.70.42.75 - pre failed: Traceback (most recent call last):
  File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 389, in <module>
    actual_end = changelog_crawl(args.brick, start, end, args)
  File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 340, in changelog_crawl
    return get_changes(brick, working_dir, log_file, start, end, args)
  File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 294, in get_changes
    parse_changelog_to_db(changelog_data, change)
  File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 216, in parse_changelog_to_db
    with open(filename) as f:
IOError: [Errno 2] No such file or directory: '/usr/var/lib/misc/glusterfsd/glusterfind/sess/vol1/226a4a26b3e4df6e10cc07aee083d8af670570f8/.history/.processing/CHANGELOG.1432030460'

10.70.42.75 - pre failed: Traceback (most recent call last):
  File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 389, in <module>
    actual_end = changelog_crawl(args.brick, start, end, args)
  File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 340, in changelog_crawl
    return get_changes(brick, working_dir, log_file, start, end, args)
  File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 294, in get_changes
    parse_changelog_to_db(changelog_data, change)
  File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 216, in parse_changelog_to_db
    with open(filename) as f:
IOError: [Errno 2] No such file or directory: '/usr/var/lib/misc/glusterfsd/glusterfind/sess/vol1/86c5809aaa8a5878c4c8b3f37184f5bb7c5c130d/.history/.processing/CHANGELOG.1432030460'

Generated output file /root/out.txt
[root@dhcp43-140 ~]# vi /root/out.txt 
[root@dhcp43-140 ~]# cat /root/out.txt 
[root@dhcp43-140 ~]# ls -l /root/out.txt 
-rw-r--r--. 1 root root 0 May 22 15:37 /root/out.txt
[root@dhcp43-140 ~]# 
[root@dhcp43-140 ~]# 
[root@dhcp43-140 ~]# gluster v info
Volume Name: vol1
Type: Distributed-Replicate
Volume ID: 44f06391-1635-4897-98c2-848e5ae92640
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 10.70.43.140:/rhs/brick1/dd
Brick2: 10.70.42.75:/rhs/brick1/dd
Brick3: 10.70.43.140:/rhs/brick2/dd
Brick4: 10.70.42.75:/rhs/brick2/dd
Options Reconfigured:
changelog.changelog: on
storage.build-pgfid: on
performance.readdir-ahead: on
[root@dhcp43-140 ~]# 
[root@dhcp43-140 ~]# 




These are the warnings displayed in the logs, along with the stack trace:

[2015-05-22 15:37:20,101] WARNING [main - 163:run_cmd_nodes] - Command pre failed in 10.70.42.75:/rhs/brick1/dd
[2015-05-22 15:37:20,101] WARNING [main - 163:run_cmd_nodes] - Command pre failed in 10.70.43.140:/rhs/brick2/dd
[2015-05-22 15:37:20,101] WARNING [main - 163:run_cmd_nodes] - Command pre failed in 10.70.42.75:/rhs/brick2/dd
[2015-05-22 15:37:20,116] WARNING [main - 118:run_cmd_nodes] - Failed to cleanup temporary file /usr/var/lib/misc/glusterfsd/glusterfind/sess/vol1/tmp_output_1
[2015-05-22 15:37:20,121] WARNING [main - 118:run_cmd_nodes] - Failed to cleanup temporary file /usr/var/lib/misc/glusterfsd/glusterfind/sess/vol1/tmp_output_3