Description of problem: Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
The bug got submitted by mistake.. apologies! This is the description: Had a 2*2 dist-rep volume, with 2 level directory depth and a couple of files in every directory. Had more than 2 sessions active for the same volume. Did creates, modify s and verified that the changes were recorded correctly in the output file. When a delete was done to one of the directory, it crashed with the below trace: 10.70.43.48 - pre failed: Traceback (most recent call last): File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 389, in <module> actual_end = changelog_crawl(args.brick, start, end, args) File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 340, in changelog_crawl return get_changes(brick, working_dir, log_file, start, end, args) File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 303, in get_changes pgfid_to_path(brick, changelog_data) File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 64, in pgfid_to_path path = symlink_gfid_to_path(brick, row[0]) File "/usr/libexec/glusterfs/glusterfind/utils.py", line 184, in symlink_gfid_to_path path_readlink = os.readlink(path) OSError: [Errno 2] No such file or directory: '/rhs/brick1/ozone/.glusterfs/e1/2f/e12f7029-e75a-4502-bac5-87f57bd78d42' Every 'pre' command subsequently run results in the same crash, and the outfile is not generated even after running a post Version-Release number: glusterfs-3.7.0-3.el6rhs.x86_64 How reproducible: Always Steps to reproduce: 1. Have a 2*2 dist rep volume 'ozone' and create 2-3 glusterfind sessions 'sess1', 'sess2', 'sess3' 2. Run the crefi command to generate a workload, of breadth 5, depth 2, and #files as 2 3. Run the glusterfind pre and post and verify that the outfile has the correct content 4. Modify (i.e., touch <filename> ) on all the files present in a particular directory, and verify the outfile after running pre and post 5. Delete one of teh entire directory (containing 4 files and 1 sub-directory) and run glutserfind pre Actual results: Step5 results in a crash with the error: No such file or directory Expected results: Step5 should log the deletes in the specified outfile Additional info: [root@dhcp43-48 ~]# [root@dhcp43-48 ~]# gluster v list ozone [root@dhcp43-48 ~]# gluster v info Volume Name: ozone Type: Distribute Volume ID: 2d22c4bb-69b4-4a33-9056-ffe7fbfe2edd Status: Started Number of Bricks: 4 Transport-type: tcp Bricks: Brick1: 10.70.43.48:/rhs/brick1/ozone Brick2: 10.70.42.28:/rhs/brick1/ozone Brick3: 10.70.43.48:/rhs/brick2/ozone Brick4: 10.70.42.28:/rhs/brick2/ozone Options Reconfigured: performance.readdir-ahead: on [root@dhcp43-48 ~]# gluster v status Status of volume: ozone Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.43.48:/rhs/brick1/ozone 49154 0 Y 11545 Brick 10.70.42.28:/rhs/brick1/ozone 49152 0 Y 3368 Brick 10.70.43.48:/rhs/brick2/ozone 49155 0 Y 11563 Brick 10.70.42.28:/rhs/brick2/ozone 49153 0 Y 3386 NFS Server on localhost 2049 0 Y 11583 NFS Server on 10.70.42.28 2049 0 Y 3406 Task Status of Volume ozone ------------------------------------------------------------------------------ There are no active volume tasks [root@dhcp43-48 ~]# [root@dhcp43-48 ~]# [root@dhcp43-48 ~]# [root@dhcp43-48 ~]# [root@dhcp43-48 ~]# [root@dhcp43-48 ~]# [root@dhcp43-48 ~]# cat /tmp/out.txt MODIFY bnew [root@dhcp43-48 ~]# glusterfind list SESSION VOLUME SESSION TIME --------------------------------------------------------------------------- sess1 ozone 2015-06-03 21:18:11 [root@dhcp43-48 ~]# [root@dhcp43-48 ~]# [root@dhcp43-48 ~]# vi /var/log/glusterfs/glusterfind/ cli.log sess1/ [root@dhcp43-48 ~]# vi /var/log/glusterfs/glusterfind/sess1/ozone/c changelog.9fa6aa8c2ba9085158689a4474160435cd998085.log changelog.log changelog.d1f95bd3646f7eb3c472a9d18eba6d875bd4e4cf.log cli.log [root@dhcp43-48 ~]# vi /var/log/glusterfs/glusterfind/sess1/ozone/c^C [root@dhcp43-48 ~]# ls /var/lib/glusterd/glusterfind/sess1/ozone/ %2Frhs%2Fbrick1%2Fozone.status %2Frhs%2Fbrick2%2Fozone.status sess1_ozone_secret.pem sess1_ozone_secret.pem.pub status [root@dhcp43-48 ~]# ls /var/lib/glusterd/glusterfind/sess1/ozone/ %2Frhs%2Fbrick1%2Fozone.status %2Frhs%2Fbrick2%2Fozone.status sess1_ozone_secret.pem sess1_ozone_secret.pem.pub status [root@dhcp43-48 ~]# glusterfind create sess2 ozone Failed to set volume option build-pgfid on: volume set: failed: One or more connected clients cannot support the feature being set. These clients need to be upgraded or disconnected before running this command again [root@dhcp43-48 ~]# glusterfind create sess2 ozone [root@dhcp43-48 ~]# # after creating files/dirs in mountpoint [root@dhcp43-48 ~]# glusterfind pre usage: glusterfind pre [-h] [--debug] [--full] [--disable-partial] [--output-prefix OUTPUT_PREFIX] [--regenerate-outfile] [-N] session volume outfile glusterfind pre: error: too few arguments [root@dhcp43-48 ~]# glusterfind pre sess2 ozone /tmp/out.txt Generated output file /tmp/out.txt [root@dhcp43-48 ~]# ls -l /tmp/out.txt -rw-r--r--. 1 root root 1155 Jun 4 15:30 /tmp/out.txt [root@dhcp43-48 ~]# cat /tmp/out.txt NEW level00 NEW level00%2F%2F5570218f%25%25ILGKQV6XMI NEW level00%2F%2Flevel10 NEW level00%2Flevel10%2F%2F5570218f%25%25O5XQ8M368A NEW level01 NEW level01%2F%2Flevel11 NEW level02 NEW level02%2F%2F55702190%25%25MAQYB6AMK9 NEW level02%2F%2Flevel12 NEW level03 NEW level03%2F%2Flevel13 NEW level04 NEW level04%2F%2F55702190%25%25PVSEY71VGQ NEW level04%2F%2Flevel14 NEW level00%2F%2F5570218f%25%25SLDHQIROY6 NEW level01%2Flevel11%2F%2F5570218f%25%25IVNTAYB6NX NEW level02%2Flevel12%2F%2F55702190%25%25YHI28RHFZZ NEW level03%2F%2F55702190%25%25EN3APUKX5M NEW level01%2F%2F5570218f%25%255A05BAN0PP NEW level03%2Flevel13%2F%2F55702190%25%25B9TBAPE7II NEW level03%2Flevel13%2F%2F55702190%25%25B60P7OG8CZ NEW level04%2F%2F55702190%25%25AGI1SV2OO7 NEW level00%2Flevel10%2F%2F5570218f%25%25TVFTRQOO4E NEW level01%2F%2F5570218f%25%2587X64WE3HB NEW level01%2Flevel11%2F%2F5570218f%25%25A2WKCH18TK NEW level02%2F%2F55702190%25%25N67O4NZUR5 NEW level02%2Flevel12%2F%2F55702190%25%2536GE0S2QCZ NEW level03%2F%2F55702190%25%25YQPVLEU4M9 NEW level04%2Flevel14%2F%2F55702190%25%25FAZ78X6QRR NEW level04%2Flevel14%2F%2F55702190%25%25P5QHISOC8L [root@dhcp43-48 ~]# cat /tmp/out.txt | wc -l 30 [root@dhcp43-48 ~]# # after deleting the entire level04 directory.. should get only 24 entries.. [root@dhcp43-48 ~]# [root@dhcp43-48 ~]# glusterfind pre sess2 ozone /tmp/out.txt --regenerate-outfile Generated output file /tmp/out.txt [root@dhcp43-48 ~]# ls -l /tmp/out.txt -rw-r--r--. 1 root root 924 Jun 4 15:32 /tmp/out.txt [root@dhcp43-48 ~]# cat /tmp/out.txt NEW level00 NEW level00%2F%2F5570218f%25%25ILGKQV6XMI NEW level00%2F%2Flevel10 NEW level00%2Flevel10%2F%2F5570218f%25%25O5XQ8M368A NEW level01 NEW level01%2F%2Flevel11 NEW level02 NEW level02%2F%2F55702190%25%25MAQYB6AMK9 NEW level02%2F%2Flevel12 NEW level03 NEW level03%2F%2Flevel13 NEW level00%2F%2F5570218f%25%25SLDHQIROY6 NEW level01%2Flevel11%2F%2F5570218f%25%25IVNTAYB6NX NEW level02%2Flevel12%2F%2F55702190%25%25YHI28RHFZZ NEW level03%2F%2F55702190%25%25EN3APUKX5M NEW level01%2F%2F5570218f%25%255A05BAN0PP NEW level03%2Flevel13%2F%2F55702190%25%25B9TBAPE7II NEW level03%2Flevel13%2F%2F55702190%25%25B60P7OG8CZ NEW level00%2Flevel10%2F%2F5570218f%25%25TVFTRQOO4E NEW level01%2F%2F5570218f%25%2587X64WE3HB NEW level01%2Flevel11%2F%2F5570218f%25%25A2WKCH18TK NEW level02%2F%2F55702190%25%25N67O4NZUR5 NEW level02%2Flevel12%2F%2F55702190%25%2536GE0S2QCZ NEW level03%2F%2F55702190%25%25YQPVLEU4M9 [root@dhcp43-48 ~]# cat /tmp/out.txt | wc -l 24 [root@dhcp43-48 ~]# [root@dhcp43-48 ~]# [root@dhcp43-48 ~]# glusterfind post sess2 ozone [root@dhcp43-48 ~]# [root@dhcp43-48 ~]# # after doing 'touch' on all the files/dirs presetn in level03 [root@dhcp43-48 ~]# [root@dhcp43-48 ~]# glusterfind pre sess2 ozone /tmp/out.txt Generated output file /tmp/out.txt [root@dhcp43-48 ~]# ls -l /tmp/out.txt -rw-r--r--. 1 root root 192 Jun 4 15:35 /tmp/out.txt [root@dhcp43-48 ~]# cat /tmp/out.txt MODIFY level03%2F55702190%25%25EN3APUKX5M MODIFY level03%2F55702190%25%25YQPVLEU4M9 MODIFY level03%2Flevel13%2F55702190%25%25B60P7OG8CZ MODIFY level03%2Flevel13%2F55702190%25%25B9TBAPE7II [root@dhcp43-48 ~]# [root@dhcp43-48 ~]# glusterfind pre sess1 ozone /tmp/out2.txt root.42.28's password: root.42.28's password: 10.70.43.48 - pre failed: /rhs/brick1/ozone Historical Changelogs not available: [Errno 2] No such file or directory 10.70.43.48 - pre failed: /rhs/brick2/ozone Historical Changelogs not available: [Errno 2] No such file or directory root.42.28's password: 10.70.42.28 - pre failed: /rhs/brick1/ozone Historical Changelogs not available: [Errno 2] No such file or directory root.42.28's password: 10.70.42.28 - pre failed: Permission denied, please try again. Permission denied, please try again. /rhs/brick2/ozone Historical Changelogs not available: [Errno 2] No such file or directory root.42.28's password: root.42.28's password: root.42.28's password: Generated output file /tmp/out2.txt [root@dhcp43-48 ~]# [root@dhcp43-48 ~]# [root@dhcp43-48 ~]# ls -l /tmp/out2.txt -rw-r--r--. 1 root root 0 Jun 4 15:36 /tmp/out2.txt [root@dhcp43-48 ~]# glusterfind pre sess1 ozone /tmp/out2.txt Post command is not run after last pre, use --regenerate-outfile [root@dhcp43-48 ~]# glusterfind pre sess1 ozone /tmp/out2.txt --regenerate-outfile root.42.28's password: root.42.28's password: 10.70.43.48 - pre failed: /rhs/brick1/ozone Historical Changelogs not available: [Errno 2] No such file or directory 10.70.43.48 - pre failed: /rhs/brick2/ozone Historical Changelogs not available: [Errno 2] No such file or directory 10.70.42.28 - pre failed: /rhs/brick1/ozone Historical Changelogs not available: [Errno 2] No such file or directory root.42.28's password: 10.70.42.28 - pre failed: Permission denied, please try again. /rhs/brick2/ozone Historical Changelogs not available: [Errno 2] No such file or directory root.42.28's password: root.42.28's password: root.42.28's password: Generated output file /tmp/out2.txt [root@dhcp43-48 ~]# ls -l /tmp/out2.txt -rw-r--r--. 1 root root 0 Jun 4 15:50 /tmp/out2.txt [root@dhcp43-48 ~]# glusterfind pre sess2 ozone /tmp/out.txt Post command is not run after last pre, use --regenerate-outfile [root@dhcp43-48 ~]# glusterfind pre sess2 ozone /tmp/out.txt --regenerate-outfile Generated output file /tmp/out.txt [root@dhcp43-48 ~]# ls -l /tmp/out.txt -rw-r--r--. 1 root root 192 Jun 4 15:51 /tmp/out.txt [root@dhcp43-48 ~]# cat /tmp/out.txt MODIFY level03%2F55702190%25%25EN3APUKX5M MODIFY level03%2F55702190%25%25YQPVLEU4M9 MODIFY level03%2Flevel13%2F55702190%25%25B60P7OG8CZ MODIFY level03%2Flevel13%2F55702190%25%25B9TBAPE7II [root@dhcp43-48 ~]# glusterfind list SESSION VOLUME SESSION TIME --------------------------------------------------------------------------- sess1 ozone 2015-06-03 21:18:11 sess2 ozone 2015-06-04 15:32:14 [root@dhcp43-48 ~]# [root@dhcp43-48 ~]# [root@dhcp43-48 ~]# glusterfind create sess3 ozone Failed to set volume option build-pgfid on: volume set: failed: One or more connected clients cannot support the feature being set. These clients need to be upgraded or disconnected before running this command again [root@dhcp43-48 ~]# glusterfind create sess3 ozone [root@dhcp43-48 ~]# glusterfind pre sess3 ozone /tmp/out3.txt Generated output file /tmp/out3.txt [root@dhcp43-48 ~]# ls -l /tmp/out3.txt -rw-r--r--. 1 root root 0 Jun 4 16:40 /tmp/out3.txt [root@dhcp43-48 ~]# vi /tmp/out3.txt [root@dhcp43-48 ~]# glusterfind pre sess3 ozone /tmp/out3.txt Post command is not run after last pre, use --regenerate-outfile [root@dhcp43-48 ~]# glusterfind pre sess3 ozone /tmp/out3.txt --regenerate-outfile Generated output file /tmp/out3.txt [root@dhcp43-48 ~]# ls -l /tmp/out3.txt -rw-r--r--. 1 root root 0 Jun 4 16:41 /tmp/out3.txt [root@dhcp43-48 ~]# glusterfind pre sess3 ozone /tmp/out3.txt --regenerate-outfile Generated output file /tmp/out3.txt [root@dhcp43-48 ~]# ls -l /tmp/out3.txt -rw-r--r--. 1 root root 192 Jun 4 16:42 /tmp/out3.txt [root@dhcp43-48 ~]# cat /tmp/out3.txt MODIFY level00%2F5570218f%25%25ILGKQV6XMI MODIFY level00%2F5570218f%25%25SLDHQIROY6 MODIFY level00%2Flevel10%2F5570218f%25%25O5XQ8M368A MODIFY level00%2Flevel10%2F5570218f%25%25TVFTRQOO4E [root@dhcp43-48 ~]# glusterfind pre sess2 ozone /tmp/out2.txt Post command is not run after last pre, use --regenerate-outfile [root@dhcp43-48 ~]# glusterfind pre sess2 ozone /tmp/out2.txt --regenerate-outfile Generated output file /tmp/out2.txt [root@dhcp43-48 ~]# ls -l /tmp/out2.txt -rw-r--r--. 1 root root 384 Jun 4 16:44 /tmp/out2.txt [root@dhcp43-48 ~]# cat /tmp/out2.txt MODIFY level03%2F55702190%25%25EN3APUKX5M MODIFY level03%2F55702190%25%25YQPVLEU4M9 MODIFY level03%2Flevel13%2F55702190%25%25B60P7OG8CZ MODIFY level03%2Flevel13%2F55702190%25%25B9TBAPE7II MODIFY level00%2F5570218f%25%25ILGKQV6XMI MODIFY level00%2F5570218f%25%25SLDHQIROY6 MODIFY level00%2Flevel10%2F5570218f%25%25O5XQ8M368A MODIFY level00%2Flevel10%2F5570218f%25%25TVFTRQOO4E [root@dhcp43-48 ~]# glusterfind post sess2 ozone [root@dhcp43-48 ~]# glusterfind post sess3 ozone [root@dhcp43-48 ~]# # after deleting the entire level03 directory [root@dhcp43-48 ~]# glusterfind pre sess2 ozone /tmp/out2.txt 10.70.43.48 - pre failed: Traceback (most recent call last): File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 389, in <module> actual_end = changelog_crawl(args.brick, start, end, args) File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 340, in changelog_crawl return get_changes(brick, working_dir, log_file, start, end, args) File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 303, in get_changes pgfid_to_path(brick, changelog_data) File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 64, in pgfid_to_path path = symlink_gfid_to_path(brick, row[0]) File "/usr/libexec/glusterfs/glusterfind/utils.py", line 184, in symlink_gfid_to_path path_readlink = os.readlink(path) OSError: [Errno 2] No such file or directory: '/rhs/brick1/ozone/.glusterfs/e1/2f/e12f7029-e75a-4502-bac5-87f57bd78d42' 10.70.43.48 - pre failed: Traceback (most recent call last): File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 389, in <module> actual_end = changelog_crawl(args.brick, start, end, args) File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 340, in changelog_crawl return get_changes(brick, working_dir, log_file, start, end, args) File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 303, in get_changes pgfid_to_path(brick, changelog_data) File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 64, in pgfid_to_path path = symlink_gfid_to_path(brick, row[0]) File "/usr/libexec/glusterfs/glusterfind/utils.py", line 184, in symlink_gfid_to_path path_readlink = os.readlink(path) OSError: [Errno 2] No such file or directory: '/rhs/brick2/ozone/.glusterfs/e1/2f/e12f7029-e75a-4502-bac5-87f57bd78d42' 10.70.42.28 - pre failed: Traceback (most recent call last): File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 389, in <module> actual_end = changelog_crawl(args.brick, start, end, args) File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 340, in changelog_crawl return get_changes(brick, working_dir, log_file, start, end, args) File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 303, in get_changes pgfid_to_path(brick, changelog_data) File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 64, in pgfid_to_path path = symlink_gfid_to_path(brick, row[0]) File "/usr/libexec/glusterfs/glusterfind/utils.py", line 184, in symlink_gfid_to_path path_readlink = os.readlink(path) OSError: [Errno 2] No such file or directory: '/rhs/brick2/ozone/.glusterfs/e1/2f/e12f7029-e75a-4502-bac5-87f57bd78d42' 10.70.42.28 - pre failed: Traceback (most recent call last): File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 389, in <module> actual_end = changelog_crawl(args.brick, start, end, args) File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 340, in changelog_crawl return get_changes(brick, working_dir, log_file, start, end, args) File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 303, in get_changes pgfid_to_path(brick, changelog_data) File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 64, in pgfid_to_path path = symlink_gfid_to_path(brick, row[0]) File "/usr/libexec/glusterfs/glusterfind/utils.py", line 184, in symlink_gfid_to_path path_readlink = os.readlink(path) OSError: [Errno 2] No such file or directory: '/rhs/brick1/ozone/.glusterfs/99/e9/99e933ec-b8e6-4336-9bb2-5affb6fa697b' Generated output file /tmp/out2.txt [root@dhcp43-48 ~]# ls -l /tmp/out2.txt -rw-r--r--. 1 root root 0 Jun 4 16:49 /tmp/out2.txt [root@dhcp43-48 ~]# cat /tmp/out2.txt [root@dhcp43-48 ~]# [root@dhcp43-48 ~]# [root@dhcp43-48 ~]# [root@dhcp43-48 ~]# [root@dhcp43-48 ~]# [root@dhcp43-48 ~]# glusterfind pre sess3 ozone /tmp/out3.txt 10.70.43.48 - pre failed: Traceback (most recent call last): File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 389, in <module> actual_end = changelog_crawl(args.brick, start, end, args) File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 340, in changelog_crawl return get_changes(brick, working_dir, log_file, start, end, args) File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 303, in get_changes pgfid_to_path(brick, changelog_data) File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 64, in pgfid_to_path path = symlink_gfid_to_path(brick, row[0]) File "/usr/libexec/glusterfs/glusterfind/utils.py", line 184, in symlink_gfid_to_path path_readlink = os.readlink(path) OSError: [Errno 2] No such file or directory: '/rhs/brick2/ozone/.glusterfs/e1/2f/e12f7029-e75a-4502-bac5-87f57bd78d42' 10.70.43.48 - pre failed: Traceback (most recent call last): File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 389, in <module> actual_end = changelog_crawl(args.brick, start, end, args) File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 340, in changelog_crawl return get_changes(brick, working_dir, log_file, start, end, args) File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 303, in get_changes pgfid_to_path(brick, changelog_data) File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 64, in pgfid_to_path path = symlink_gfid_to_path(brick, row[0]) File "/usr/libexec/glusterfs/glusterfind/utils.py", line 184, in symlink_gfid_to_path path_readlink = os.readlink(path) OSError: [Errno 2] No such file or directory: '/rhs/brick1/ozone/.glusterfs/e1/2f/e12f7029-e75a-4502-bac5-87f57bd78d42' 10.70.42.28 - pre failed: Traceback (most recent call last): File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 389, in <module> actual_end = changelog_crawl(args.brick, start, end, args) File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 340, in changelog_crawl return get_changes(brick, working_dir, log_file, start, end, args) File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 303, in get_changes pgfid_to_path(brick, changelog_data) File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 64, in pgfid_to_path path = symlink_gfid_to_path(brick, row[0]) File "/usr/libexec/glusterfs/glusterfind/utils.py", line 184, in symlink_gfid_to_path path_readlink = os.readlink(path) OSError: [Errno 2] No such file or directory: '/rhs/brick1/ozone/.glusterfs/99/e9/99e933ec-b8e6-4336-9bb2-5affb6fa697b' 10.70.42.28 - pre failed: Traceback (most recent call last): File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 389, in <module> actual_end = changelog_crawl(args.brick, start, end, args) File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 340, in changelog_crawl return get_changes(brick, working_dir, log_file, start, end, args) File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 303, in get_changes pgfid_to_path(brick, changelog_data) File "/usr/libexec/glusterfs/glusterfind/changelog.py", line 64, in pgfid_to_path path = symlink_gfid_to_path(brick, row[0]) File "/usr/libexec/glusterfs/glusterfind/utils.py", line 184, in symlink_gfid_to_path path_readlink = os.readlink(path) OSError: [Errno 2] No such file or directory: '/rhs/brick2/ozone/.glusterfs/e1/2f/e12f7029-e75a-4502-bac5-87f57bd78d42' Generated output file /tmp/out3.txt [root@dhcp43-48 ~]# 2.
Just to note: When glusterfind pre results in a crash, it does says that the outfile is generated - which is an empty file. The subsequent post command does update the status file at the $GLUSTERD_WORKDIR. So it is expected that the 'pre' command given after running the post, should work, but results in the same crash, and an empty outfile.
Changelog failed here to record the deleted path for RMDIR. E99e933ec-b8e6-4336-9bb2-5affb6fa697b^@6^@e12f7029-e75a-4502-bac5-87f57bd78d42/level13^@^@^@Ee12f7029-e75a-4502-bac5-87f57bd78d42^@6^@00000000-0000-0000-0000-000000000001/level03^@^@^@ Need to check why Changelog not recorded the deleted path of directory. Also Fix is required in glusterfind not to raise traceback when it don't find file/dir.
glusterfind changes for this bug and BZ 1224033 is same.
Tested and verified this on the build glusterfs-3.7.1-4.el6rhs.x86_64 Did not hit a crash when directory structure(s) or file(s) were deleted. Moving this to verified in 3.1 everglades. Pasted below are the logs. Functional testing in and around this fix has been uploaded at https://polarion.engineering.redhat.com/polarion/testrun-attachment/RHG3/glusterfs-3_7_1_3_RHEL6_7_FUSE/RHG3-5400_Logs_6.7_3.7.1-3_output_file_validation.odt [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# glusterfind list SESSION VOLUME SESSION TIME --------------------------------------------------------------------------- sesso3 ozone 2015-06-18 16:27:30 sesso1 ozone 2015-06-23 20:43:34 cross3s1 cross3 2015-06-23 23:30:16 cross3s3 cross3 2015-06-23 17:55:28 cross3s2 cross3 2015-06-23 18:06:38 sesso2 ozone 2015-06-20 00:14:05 sesso4 ozone 2015-06-18 16:27:38 [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# glusterfind pre sesso1 ozone /tmp/outo1.txt Post command is not run after last pre, use --regenerate-outfile [root@dhcp43-191 ~]# glusterfind post sesso1 ozone Session sesso1 with volume ozone updated [root@dhcp43-191 ~]# glusterfind pre sesso1 ozone /tmp/outo1.txt 10.70.42.30 - pre failed: /rhs/thinbrick2/ozone Historical Changelogs not available: [Errno 2] No such file or directory 10.70.42.147 - pre failed: /rhs/thinbrick2/ozone Historical Changelogs not available: [Errno 2] No such file or directory Generated output file /tmp/outo1.txt [root@dhcp43-191 ~]# vi /tmp/outo1.txt [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# cat /tmp/outo1.txt | wc -l 37 [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# # after deleting another directory [root@dhcp43-191 ~]# glusterfind post sesso1 ozone Session sesso1 with volume ozone updated [root@dhcp43-191 ~]# glusterfind pre sesso1 ozone /tmp/outo1.txt 10.70.42.30 - pre failed: [2015-06-25 12:46:31.542904] I [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2 [2015-06-25 12:46:31.542935] I [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 /rhs/thinbrick2/ozone Historical Changelogs not available: [Errno 2] No such file or directory 10.70.42.147 - pre failed: /rhs/thinbrick2/ozone Historical Changelogs not available: [Errno 2] No such file or directory Generated output file /tmp/outo1.txt [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# vi /tmp/outo1.txt [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# glusterfind pre sesso1 ozone /tmp/outo1.txt --regenerate-outfile 10.70.42.30 - pre failed: [2015-06-25 12:47:08.360553] I [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2015-06-25 12:47:08.361238] I [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2 [2015-06-25 12:47:08.361422] I [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 3 [2015-06-25 12:47:08.361499] I [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 4 /rhs/thinbrick2/ozone Historical Changelogs not available: [Errno 2] No such file or directory 10.70.42.147 - pre failed: [2015-06-25 12:47:08.546771] I [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2015-06-25 12:47:08.546954] I [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2 [2015-06-25 12:47:08.547067] I [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 3 [2015-06-25 12:47:08.547334] I [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 4 /rhs/thinbrick2/ozone Historical Changelogs not available: [Errno 2] No such file or directory Generated output file /tmp/outo1.txt [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# vi /tmp/outo1.txt [root@dhcp43-191 ~]# cat /tmp/outo1.txt | wc -l 27 [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# glusterfind list SESSION VOLUME SESSION TIME --------------------------------------------------------------------------- sesso3 ozone 2015-06-18 16:27:30 sesso1 ozone 2015-06-25 18:15:03 cross3s1 cross3 2015-06-23 23:30:16 cross3s3 cross3 2015-06-23 17:55:28 cross3s2 cross3 2015-06-23 18:06:38 sesso2 ozone 2015-06-20 00:14:05 sesso4 ozone 2015-06-18 16:27:38 [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# glusterfind pre sesso1 ozone /tmp/outo1.txt Post command is not run after last pre, use --regenerate-outfile [root@dhcp43-191 ~]# glusterfind post sesso1 ozone Session sesso1 with volume ozone updated [root@dhcp43-191 ~]# glusterfind pre sesso1 ozone /tmp/outo1.txt 10.70.42.30 - pre failed: /rhs/thinbrick2/ozone Historical Changelogs not available: [Errno 2] No such file or directory 10.70.42.147 - pre failed: [2015-06-25 12:47:51.300569] I [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2015-06-25 12:47:51.302629] I [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 3 [2015-06-25 12:47:51.302759] I [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2 [2015-06-25 12:47:51.302787] I [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 4 /rhs/thinbrick2/ozone Historical Changelogs not available: [Errno 2] No such file or directory Generated output file /tmp/outo1.txt [root@dhcp43-191 ~]# vi /tmp/outo1.txt [root@dhcp43-191 ~]# ls -l /tmp/outo1.txt -rw-r--r--. 1 root root 0 Jun 25 18:17 /tmp/outo1.txt [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# glusterfind pre sesso2 ozone /tmp/outo2.txt 10.70.42.30 - pre failed: /rhs/thinbrick2/ozone Historical Changelogs not available: [Errno 2] No such file or directory 10.70.42.147 - pre failed: /rhs/thinbrick2/ozone Historical Changelogs not available: [Errno 2] No such file or directory Generated output file /tmp/outo2.txt [root@dhcp43-191 ~]# vi /tmp/outo2.txt [root@dhcp43-191 ~]# glusterfind pre sesso2 ozone /tmp/outo2.txt Post command is not run after last pre, use --regenerate-outfile [root@dhcp43-191 ~]# glusterfind pre sesso2 ozone /tmp/outo2.txt --regenerate-outfile 10.70.42.30 - pre failed: /rhs/thinbrick2/ozone Historical Changelogs not available: [Errno 2] No such file or directory 10.70.42.147 - pre failed: /rhs/thinbrick2/ozone Historical Changelogs not available: [Errno 2] No such file or directory Generated output file /tmp/outo2.txt [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# vi /tmp/outo2.txt [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# glusterfind pre sesso3 ozone /tmp/outo3.txt 10.70.42.30 - pre failed: [2015-06-25 12:50:35.347455] I [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 4 /rhs/thinbrick2/ozone Historical Changelogs not available: [Errno 2] No such file or directory 10.70.42.147 - pre failed: /rhs/thinbrick2/ozone Historical Changelogs not available: [Errno 2] No such file or directory Generated output file /tmp/outo3.txt [root@dhcp43-191 ~]# vi /tmp/outo3.txt [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# vi /var/log/glusterfs/glusterfind/ ^%#@*/ cli.log cross3s2/ demo/ fdjksl/ nosession/ ozone/ session/ sesso1/ sesso3/ sesso5/ sess_vol1/ 5543/ cross3s1/ cross3s3/ demosess/ fds/ novolume/ sess_demo/ sessn1/ sesso2/ sesso4/ sesso6/ [root@dhcp43-191 ~]# vi /var/log/glusterfs/glusterfind/sesso1/ cli.log ozone/ [root@dhcp43-191 ~]# vi /var/log/glusterfs/glusterfind/sesso1/ozone/c changelog.08da96af79321aeb6981b439160e3de184fc358b.log changelog.log changelog.8d99046d2c04c2fe05e40b94244201e8fefd9979.log cli.log [root@dhcp43-191 ~]# vi /var/log/glusterfs/glusterfind/sesso1/ozone/cli.log [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# glusterfind list SESSION VOLUME SESSION TIME --------------------------------------------------------------------------- sesso3 ozone 2015-06-18 16:27:30 sesso1 ozone 2015-06-25 18:16:52 cross3s1 cross3 2015-06-23 23:30:16 cross3s3 cross3 2015-06-23 17:55:28 cross3s2 cross3 2015-06-23 18:06:38 sesso2 ozone 2015-06-20 00:14:05 sesso4 ozone 2015-06-18 16:27:38 [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# glusterfind post cross3s1 cross3 Session cross3s1 with volume cross3 updated [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# glusterfind pre cross3s1 cross3 /tmp/out3 out31.txt out32.txt [root@dhcp43-191 ~]# glusterfind pre cross3s1 cross3 /tmp/out31.txt Generated output file /tmp/out31.txt [root@dhcp43-191 ~]# vi /tmp/out31.txt [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# glusterfind post cross3s1 cross3 Session cross3s1 with volume cross3 updated [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# glusterfind pre cross3s1 cross3 /tmp/out31.txt Generated output file /tmp/out31.txt [root@dhcp43-191 ~]# vi /tmp/out31.txt [root@dhcp43-191 ~]# cat /tmp/out31.txt DELETE test2 [root@dhcp43-191 ~]# glusterfind pre cross3s2 cross3 /tmp/out32.txt Generated output file /tmp/out32.txt [root@dhcp43-191 ~]# vi /tmp/out31.txt [root@dhcp43-191 ~]# vi /tmp/out32.txt [root@dhcp43-191 ~]# vi /tmp/out32.txt [root@dhcp43-191 ~]# cat /tmp/out32.txt RENAME a dir1%2Fdir2%2Fa DELETE dir1%2Fdir2%2Fanew DELETE dir1%2Fdir2%2Ffile123 DELETE dir1 DELETE dir1%2Fdir2%2Ffile456 DELETE test2 [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# glusterfind post cross3s2 cross3 Session cross3s2 with volume cross3 updated [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# glusterfind pre cross3s2 cross3 /tmp/out32.txt Generated output file /tmp/out32.txt [root@dhcp43-191 ~]# cat /tmp/out32.txt NEW dir1 [root@dhcp43-191 ~]# glusterfind pre cross3s2 cross3 /tmp/out32.txt --regenerate-outfiel usage: glusterfind [-h] {pre,create,list,post,delete} ... glusterfind: error: unrecognized arguments: --regenerate-outfiel [root@dhcp43-191 ~]# glusterfind pre cross3s2 cross3 /tmp/out32.txt --regenerate-outfile Generated output file /tmp/out32.txt [root@dhcp43-191 ~]# cat /tmp/out32.txt [root@dhcp43-191 ~]# glusterfind pre cross3s2 cross3 /tmp/out32.txt --regenerate-outfile Generated output file /tmp/out32.txt [root@dhcp43-191 ~]# cat /tmp/out32.txt [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# # after creating a directory structure of depth 10 [root@dhcp43-191 ~]# glusterfind pre cross3s2 cross3 /tmp/out32.txt --regenerate-outfile Generated output file /tmp/out32.txt [root@dhcp43-191 ~]# cat /tmp/out32.txt NEW dir1 NEW dir1%2F%2Fdir2 NEW dir1%2Fdir2%2F%2Fdir3 NEW dir1%2Fdir2%2Fdir3%2F%2Fdir4 NEW dir1%2Fdir2%2Fdir3%2Fdir4%2F%2Fdir5 NEW dir1%2Fdir2%2Fdir3%2Fdir4%2Fdir5%2F%2Fdir6 NEW dir1%2Fdir2%2Fdir3%2Fdir4%2Fdir5%2Fdir6%2F%2Fdir7 NEW dir1%2Fdir2%2Fdir3%2Fdir4%2Fdir5%2Fdir6%2Fdir7%2F%2Fdir8 NEW dir1%2Fdir2%2Fdir3%2Fdir4%2Fdir5%2Fdir6%2Fdir7%2Fdir8%2F%2Fdir9 NEW dir1%2Fdir2%2Fdir3%2Fdir4%2Fdir5%2Fdir6%2Fdir7%2Fdir8%2Fdir9%2F%2Fdir10 [root@dhcp43-191 ~]# glusterfind post cross3s2 cross3 Session cross3s2 with volume cross3 updated [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# # after deleting the directory structure [root@dhcp43-191 ~]# glusterfind pre cross3s2 cross3 /tmp/out32.txt Generated output file /tmp/out32.txt [root@dhcp43-191 ~]# cat /tmp/out32.txt DELETE dir1%2Fdir2%2Fdir3%2Fdir4%2Fdir5%2Fdir6%2Fdir7%2Fdir8%2Fdir9%2Fdir10 DELETE dir1%2Fdir2%2Fdir3%2Fdir4%2Fdir5%2Fdir6%2Fdir7%2Fdir8%2Fdir9 DELETE dir1%2Fdir2%2Fdir3%2Fdir4%2Fdir5%2Fdir6%2Fdir7%2Fdir8 DELETE dir1%2Fdir2%2Fdir3%2Fdir4%2Fdir5%2Fdir6%2Fdir7 DELETE dir1%2Fdir2%2Fdir3%2Fdir4%2Fdir5%2Fdir6 DELETE dir1%2Fdir2%2Fdir3%2Fdir4%2Fdir5 DELETE dir1%2Fdir2%2Fdir3%2Fdir4 DELETE dir1%2Fdir2%2Fdir3 DELETE dir1%2Fdir2 DELETE dir1 [root@dhcp43-191 ~]# glusterfind pre cross3s1 cross3 /tmp/out31.txt Post command is not run after last pre, use --regenerate-outfile [root@dhcp43-191 ~]# glusterfind post cross3s1 cross3 Session cross3s1 with volume cross3 updated [root@dhcp43-191 ~]# glusterfind pre cross3s1 cross3 /tmp/out31.txt Generated output file /tmp/out31.txt [root@dhcp43-191 ~]# cat /tmp/out31.txt [root@dhcp43-191 ~]# cat /tmp/out31.txt [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# glusterfind post cross3s1 cross3 Session cross3s1 with volume cross3 updated [root@dhcp43-191 ~]# glusterfind post cross3s2 cross3 Session cross3s2 with volume cross3 updated [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# glusterfind pre cross3s1 cross3 /tmp/out31.txt Generated output file /tmp/out31.txt [root@dhcp43-191 ~]# cat /tmp/out31.txt DELETE dir2 DELETE dir2%2Ftest1copy [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# rpm -qa | grep glusterfs glusterfs-libs-3.7.1-4.el6rhs.x86_64 glusterfs-api-3.7.1-4.el6rhs.x86_64 glusterfs-client-xlators-3.7.1-4.el6rhs.x86_64 glusterfs-fuse-3.7.1-4.el6rhs.x86_64 glusterfs-cli-3.7.1-4.el6rhs.x86_64 glusterfs-3.7.1-4.el6rhs.x86_64 glusterfs-server-3.7.1-4.el6rhs.x86_64 [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# gluster v list cross3 gluster_shared_storage nash ozone [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# [root@dhcp43-191 ~]# gluster v info ozone Volume Name: ozone Type: Distributed-Replicate Volume ID: 9ef1ace8-505d-4d97-aa23-4296aa685f76 Status: Started Number of Bricks: 4 x 2 = 8 Transport-type: tcp Bricks: Brick1: 10.70.43.191:/rhs/thinbrick1/ozone Brick2: 10.70.42.202:/rhs/thinbrick1/ozone Brick3: 10.70.43.191:/rhs/thinbrick2/ozone Brick4: 10.70.42.202:/rhs/thinbrick2/ozone Brick5: 10.70.42.30:/rhs/thinbrick1/ozone Brick6: 10.70.42.147:/rhs/thinbrick1/ozone Brick7: 10.70.42.30:/rhs/thinbrick2/ozone Brick8: 10.70.42.147:/rhs/thinbrick2/ozone Options Reconfigured: performance.readdir-ahead: on storage.build-pgfid: on changelog.changelog: on changelog.capture-del-path: on [root@dhcp43-191 ~]# gluster v info cross34 Volume cross34 does not exist [root@dhcp43-191 ~]# gluster v info cross3 Volume Name: cross3 Type: Distributed-Replicate Volume ID: 81de5a1c-24ac-44fa-a9ce-7691d9b308a0 Status: Started Number of Bricks: 2 x 3 = 6 Transport-type: tcp Bricks: Brick1: 10.70.43.191:/rhs/thinbrick1/cross3 Brick2: 10.70.42.202:/rhs/thinbrick1/cross3 Brick3: 10.70.42.30:/rhs/thinbrick1/cross3 Brick4: 10.70.43.191:/rhs/thinbrick2/cross3 Brick5: 10.70.42.202:/rhs/thinbrick2/cross3 Brick6: 10.70.42.30:/rhs/thinbrick2/cross3 Options Reconfigured: changelog.capture-del-path: on changelog.changelog: on storage.build-pgfid: on performance.readdir-ahead: on [root@dhcp43-191 ~]#
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2015-1495.html