Bug 1233622

Summary:	[Backup]: Glusterfind pre fails in a dist-rep volume when one of the nodes is down
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Sweta Anandpara <sanandpa>
Component:	glusterfind	Assignee:	Srijan Sivakumar <ssivakum>
Status:	CLOSED WONTFIX	QA Contact:	Sweta Anandpara <sanandpa>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	rhgs-3.1	CC:	avishwan, khiremat, mchangir, rhs-bugs, ssivakum
Target Milestone:	---	Keywords:	Reopened, ZStream
Target Release:	---	Flags:	ssivakum: needinfo-
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-04-07 14:31:20 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1223636

Description Sweta Anandpara 2015-06-19 09:57:55 UTC

Description of problem:

In my 4 node cluster setup, had a 4*2 dist-rep volume 'ozone'. Got node3 down (which had 2 data bricks belonging to ozone), and created a directory tree structure and ran glusterfind pre on session 'sesso1'. That displayed the error 'no route to host', and gave an output file which was empty. Executed the same command on another session 'sesso2' and that failed with 'no route to host' error. BUT, that populated the output file correctly, as expected. 

When a node is down, ideally 
1. It should not have impacted the output of glusterfind. 
2. Error messages should not have been seen on the console stating 'pre failed.. connection timed out.. no route to host'.
3. Whatever behaviour is expected, should be the same across all glusterfind sessions (of the same volume)

Version-Release number of selected component (if applicable):
glusterfs-3.7.1-4.el6rhs.x86_64

How reproducible: 1:1


Steps that I followed:
1. Have a 4 node cluster, with node1, node2, node3 and node4. Create a 4*2 distribute replicate volume 'ozone', with node1 and node3 having the data bricks and node2 and node4 having the replica bricks.
2. Create glusterfind session 'sesso1' and 'sesso2' for 'ozone', and mount it using fuse/nfs
3. Create a data set at the mountpoint and run glusterfind pre and post a couple of times and verify the output file, regularly at 'sesso1' and intermittently at 'sesso2'
4. Halt node3
5. Create a directory tree structure at the mountpoint (>100 entries) and run glusterfind pre and post on 'sesso1' 
6. Do a couple of changes and again run glusterfind pre and post on 'sesso1'
7. Run glusterfind pre and post on 'sesso2'


Actual results:

Step5 gives the below output, with an *empty* output file.

[root@dhcp43-191 ~]# glusterfind pre sesso1 ozone /tmp/outo1.txt --regenerate-outfile
10.70.42.30 - pre failed: ssh: connect to host 10.70.42.30 port 22: No route to host

10.70.42.30 - pre failed: ssh: connect to host 10.70.42.30 port 22: No route to host

10.70.42.30 - cleanup failed: ssh: connect to host 10.70.42.30 port 22: No route to host

10.70.42.30 - cleanup failed: ssh: connect to host 10.70.42.30 port 22: No route to host

Generated output file /tmp/outo1.txt
[root@dhcp43-191 ~]# cat /tmp/outo1.txt 
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# ls -l /tmp/outo1.txt 
-rw-r--r--. 1 root root 0 Jun 19 20:28 /tmp/outo1.txt
[root@dhcp43-191 ~]# 


Step6 gives the same output as mentioned above

Step7 gives the below output, but *does* populate the output file as expected

[root@dhcp43-191 ~]# glusterfind pre sesso2 ozone /tmp/outo2.txt 
10.70.42.30 - pre failed: ssh: connect to host 10.70.42.30 port 22: No route to host

10.70.42.30 - pre failed: ssh: connect to host 10.70.42.30 port 22: No route to host

10.70.42.30 - cleanup failed: ssh: connect to host 10.70.42.30 port 22: No route to host

10.70.42.30 - cleanup failed: ssh: connect to host 10.70.42.30 port 22: No route to host

Generated output file /tmp/outo2.txt
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# ls -l /tmp/outo2.txt 
-rw-r--r--. 1 root root 10233 Jun 19 20:28 /tmp/outo2.txt
[root@dhcp43-191 ~]# vi /tmp/outo2.txt 
[root@dhcp43-191 ~]# 


Expected results:
Step5 and step6 should NOT have displayed errors on the console. Also, the output file should have got correctly populated.

Step7 should not have displayed the errors on the console.

Additional info:

[root@dhcp42-202 ~]# 
[root@dhcp42-202 ~]# gluster peer status
Number of Peers: 3

Hostname: 10.70.42.30
Uuid: 38189182-2e48-4f9d-a342-f595389bf1bd
State: Peer in Cluster (Connected)

Hostname: 10.70.42.147
Uuid: dc3c70a2-aa43-4bf1-ba4d-1272616b69c1
State: Peer in Cluster (Connected)

Hostname: 10.70.43.191
Uuid: eff916e2-3d5f-43ae-abab-05bbcbdf473c
State: Peer in Cluster (Connected)
[root@dhcp42-202 ~]# rpm -qa | grep glusterfs
glusterfs-libs-3.7.1-4.el6rhs.x86_64
glusterfs-api-3.7.1-4.el6rhs.x86_64
glusterfs-client-xlators-3.7.1-4.el6rhs.x86_64
glusterfs-fuse-3.7.1-4.el6rhs.x86_64
glusterfs-cli-3.7.1-4.el6rhs.x86_64
glusterfs-3.7.1-4.el6rhs.x86_64
glusterfs-server-3.7.1-4.el6rhs.x86_64
[root@dhcp42-202 ~]#

Comment 2 Sweta Anandpara 2015-06-19 10:03:23 UTC

Pasting everything that I did, since morning:

[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# gluster v info
 
Volume Name: gluster_shared_storage
Type: Replicate
Volume ID: ced3ec30-654b-4bf5-956b-9e99bc51d445
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 10.70.43.191:/rhs/brick1/gss
Brick2: 10.70.42.202:/rhs/brick1/gss
Brick3: 10.70.42.30:/rhs/brick1/gss
Options Reconfigured:
performance.readdir-ahead: on
 
Volume Name: ozone
Type: Distributed-Replicate
Volume ID: 9ef1ace8-505d-4d97-aa23-4296aa685f76
Status: Started
Number of Bricks: 4 x 2 = 8
Transport-type: tcp
Bricks:
Brick1: 10.70.43.191:/rhs/thinbrick1/ozone
Brick2: 10.70.42.202:/rhs/thinbrick1/ozone
Brick3: 10.70.43.191:/rhs/thinbrick2/ozone
Brick4: 10.70.42.202:/rhs/thinbrick2/ozone
Brick5: 10.70.42.30:/rhs/thinbrick1/ozone
Brick6: 10.70.42.147:/rhs/thinbrick1/ozone
Brick7: 10.70.42.30:/rhs/thinbrick2/ozone
Brick8: 10.70.42.147:/rhs/thinbrick2/ozone
Options Reconfigured:
changelog.capture-del-path: on
changelog.changelog: on
storage.build-pgfid: on
performance.readdir-ahead: on
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# glusterfind list
SESSION                   VOLUME                    SESSION TIME             
---------------------------------------------------------------------------
sesso3                    ozone                     2015-06-18 16:27:30      
sesso1                    ozone                     2015-06-18 21:05:19      
sesso2                    ozone                     2015-06-18 20:00:38      
sesso4                    ozone                     2015-06-18 16:27:38      
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# glusterfind post sesso1 ozone
Session sesso1 with volume ozone updated
[root@dhcp43-191 ~]# glusterfind post sesso2 ozone
Session sesso2 with volume ozone updated
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# glusterfind pre sesso1 ozone /tmp/outo1.txt 
Generated output file /tmp/outo1.txt
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# cat /tmp/outo1.txt 
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# after doing chown as well as chgrp to level01
-bash: after: command not found
[root@dhcp43-191 ~]# glusterfind post sesso2 ozone
Pre script is not run
[root@dhcp43-191 ~]# glusterfind post sesso2 ozone
Pre script is not run
[root@dhcp43-191 ~]# glusterfind pre sesso1 ozone /tmp/outo1.txt 
Post command is not run after last pre, use --regenerate-outfile
[root@dhcp43-191 ~]# glusterfind post sesso1 ozone
Session sesso1 with volume ozone updated
[root@dhcp43-191 ~]# glusterfind pre sesso1 ozone /tmp/outo1.txt 
Generated output file /tmp/outo1.txt
[root@dhcp43-191 ~]# cat /tmp/outo1.txt 
MODIFY level01%2F 
MODIFY level20%2F 
[root@dhcp43-191 ~]# glusterfind post sesso1 ozone
Session sesso1 with volume ozone updated
[root@dhcp43-191 ~]# # after touching a symbolic link file
[root@dhcp43-191 ~]# glusterfind pre sesso1 ozone /tmp/outo1.txt 
Generated output file /tmp/outo1.txt
[root@dhcp43-191 ~]# cat /tmp/outo1.txt 
MODIFY level20%2F 
[root@dhcp43-191 ~]# # after renaming a symbolic link file
[root@dhcp43-191 ~]# glusterfind post sesso1 ozone
Session sesso1 with volume ozone updated
[root@dhcp43-191 ~]# glusterfind pre sesso1 ozone /tmp/outo1.txt 
Generated output file /tmp/outo1.txt
[root@dhcp43-191 ~]# cat /tmp/outo1.txt 
RENAME level20_sln level20_sln2
[root@dhcp43-191 ~]# glusterfind post sesso1 ozone
Session sesso1 with volume ozone updated
[root@dhcp43-191 ~]# glusterfind pre sesso1 ozone /tmp/outo1.txt 
Generated output file /tmp/outo1.txt
[root@dhcp43-191 ~]# cat /tmp/outo1.txt 
NEW level20%2F%2Flevel01 
[root@dhcp43-191 ~]# glusterfind post sesso1 ozone
Session sesso1 with volume ozone updated
[root@dhcp43-191 ~]# glusterfind pre sesso1 ozone /tmp/outo1.txt 
Generated output file /tmp/outo1.txt
[root@dhcp43-191 ~]# cat /tmp/outo1.txt 
NEW a 
[root@dhcp43-191 ~]# glusterfind pre sesso1 ozone /tmp/outo1.txt --regenerate-outfile
Generated output file /tmp/outo1.txt
[root@dhcp43-191 ~]# cat /tmp/outo1.txt 
NEW a 
NEW a_symlink 
[root@dhcp43-191 ~]# glusterfind pre sesso1 ozone /tmp/outo1.txt --regenerate-outfile
Generated output file /tmp/outo1.txt
[root@dhcp43-191 ~]# cat /tmp/outo1.txt 
NEW a 
NEW a_symlink 
[root@dhcp43-191 ~]# glusterfind post sesso1 ozone
Session sesso1 with volume ozone updated
[root@dhcp43-191 ~]# glusterfind pre sesso1 ozone /tmp/outo1.txt 
Generated output file /tmp/outo1.txt
[root@dhcp43-191 ~]# cat /tmp/outo1.txt 
NEW b 
NEW a_symlink_to_b 
[root@dhcp43-191 ~]# glusterfind post sesso1 ozone
Session sesso1 with volume ozone updated
[root@dhcp43-191 ~]# glusterfind pre sesso1 ozone /tmp/outo1.txt 
Generated output file /tmp/outo1.txt
[root@dhcp43-191 ~]# cat /tmp/outo1.txt 
DELETE a_symlink_to_b 
[root@dhcp43-191 ~]# glusterfind pre sesso1 ozone /tmp/outo1.txt --regenerate-outfile
Generated output file /tmp/outo1.txt
[root@dhcp43-191 ~]# cat /tmp/outo1.txt 
DELETE a_symlink 
DELETE a_symlink_to_b 
[root@dhcp43-191 ~]# glusterfind pre sesso1 ozone /tmp/outo1.txt --regenerate-outfile
Generated output file /tmp/outo1.txt
[root@dhcp43-191 ~]# cat /tmp/outo1.txt 
DELETE a_symlink 
DELETE a_symlink_to_b 
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# # after changing recursively the group ownership of level31
[root@dhcp43-191 ~]# glusterfind post sesso1 ozone
Session sesso1 with volume ozone updated
[root@dhcp43-191 ~]# glusterfind pre sesso1 ozone /tmp/outo1.txt 
Generated output file /tmp/outo1.txt
[root@dhcp43-191 ~]# cat /tmp/outo1.txt 
MODIFY level01%2Flevel11%2Flevel21%2Flevel31%2F5582c256%25%25MPIBDVQJQ4 
MODIFY level01%2Flevel11%2Flevel21%2Flevel31%2F5582c255%25%25WRBYSGT7PZ 
MODIFY level01%2Flevel11%2Flevel21%2Flevel31%2F 
MODIFY level01%2Flevel11%2Flevel21%2Flevel31%2F5582c256%25%251QNX332SKS 
MODIFY level01%2Flevel11%2Flevel21%2Flevel31%2F5582c255%25%25OQ65TSMRUD 
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# # after creating a hardlink file
[root@dhcp43-191 ~]# glusterfind post sesso1 ozone
Session sesso1 with volume ozone updated
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# glusterfind pre sesso1 ozone /tmp/outo1.txt 
Generated output file /tmp/outo1.txt
[root@dhcp43-191 ~]# cat /tmp/outo1.txt 
NEW level01%2Flevel11%2F%2Fhardlink_file 
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# # after rename and simulateneous movement of hardlink file
[root@dhcp43-191 ~]# glusterfind post sesso1 ozone
Session sesso1 with volume ozone updated
[root@dhcp43-191 ~]# glusterfind pre sesso1 ozone /tmp/outo1.txt 
10.70.42.30 - pre failed: 
^[[A^[[AGenerated output file /tmp/outo1.txt
[root@dhcp43-191 ~]# glusterfind post sesso1 ozone --regenerate-outfile
usage: glusterfind [-h] {pre,create,list,post,delete} ...
glusterfind: error: unrecognized arguments: --regenerate-outfile
[root@dhcp43-191 ~]# glusterfind pre sesso1 ozone /tmp/outo1.txt --regenerate-outfile
Generated output file /tmp/outo1.txt
[root@dhcp43-191 ~]# cat /tmp/outo1.txt 
RENAME level01%2Flevel11%2F%2Fhardlink_file level01%2F%2Fhardlink
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# cd
[root@dhcp43-191 ~]# gluster peer status
Number of Peers: 3

Hostname: 10.70.42.202
Uuid: 2abb8dbb-48db-4c26-97bb-1ae1e11964f1
State: Peer in Cluster (Connected)

Hostname: 10.70.42.30
Uuid: 38189182-2e48-4f9d-a342-f595389bf1bd
State: Peer in Cluster (Connected)

Hostname: 10.70.42.147
Uuid: dc3c70a2-aa43-4bf1-ba4d-1272616b69c1
State: Peer in Cluster (Connected)
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# 


##########  changed the glusterfs build from 3.7.1-3 to 3.7.1-4   ##########

[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# service glusterd stop
[root@dhcp43-191 ~]#                                       [  OK  ]
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# pkill glusterfs
[root@dhcp43-191 ~]# pkill glusterfsd
[root@dhcp43-191 ~]# pkill gsyncd.py
[root@dhcp43-191 ~]# yum remove gluster
Loaded plugins: product-id, security, subscription-manager
Setting up Remove Process
No Match for argument: gluster
epel/metalink                                                                                                                                                          | 4.1 kB     00:00     
rhel-6-server-rpms                                                                                                                                                     | 3.7 kB     00:00     
rhel-6-server-rpms/primary_db                                                                                                                                          |  32 MB     03:52     
rhel-scalefs-for-rhel-6-server-rpms                                                                                                                                    | 4.6 kB     00:00     
rhs-3-for-rhel-6-server-rpms                                                                                                                                           | 3.1 kB     00:00     
rhsqe-repo.lab.eng.blr.redhat.com_rhgs3.1-server-latest-el6_                                                                                                           | 2.9 kB     00:00     
rhsqe-repo.lab.eng.blr.redhat.com_rhgs3.1-server-latest-el6_/primary_db                                                                                                |  13 kB     00:00     
No Packages marked for removal
[root@dhcp43-191 ~]# yum remove glusterfs
Loaded plugins: product-id, security, subscription-manager
Setting up Remove Process
Resolving Dependencies
--> Running transaction check
---> Package glusterfs.x86_64 0:3.7.1-3.el6rhs will be erased
--> Processing Dependency: glusterfs = 3.7.1-3.el6rhs for package: glusterfs-api-3.7.1-3.el6rhs.x86_64
--> Processing Dependency: glusterfs = 3.7.1-3.el6rhs for package: glusterfs-server-3.7.1-3.el6rhs.x86_64
--> Processing Dependency: glusterfs = 3.7.1-3.el6rhs for package: glusterfs-fuse-3.7.1-3.el6rhs.x86_64
--> Running transaction check
---> Package glusterfs-api.x86_64 0:3.7.1-3.el6rhs will be erased
---> Package glusterfs-fuse.x86_64 0:3.7.1-3.el6rhs will be erased
---> Package glusterfs-server.x86_64 0:3.7.1-3.el6rhs will be erased
--> Finished Dependency Resolution

Dependencies Resolved

==============================================================================================================================================================================================
 Package                              Arch                       Version                              Repository                                                                         Size
==============================================================================================================================================================================================
Removing:
 glusterfs                            x86_64                     3.7.1-3.el6rhs                       @rhsqe-repo.lab.eng.blr.redhat.com_rhgs3.1-server-latest-el6_                     1.5 M
Removing for dependencies:
 glusterfs-api                        x86_64                     3.7.1-3.el6rhs                       @rhsqe-repo.lab.eng.blr.redhat.com_rhgs3.1-server-latest-el6_                     132 k
 glusterfs-fuse                       x86_64                     3.7.1-3.el6rhs                       @rhsqe-repo.lab.eng.blr.redhat.com_rhgs3.1-server-latest-el6_                     304 k
 glusterfs-server                     x86_64                     3.7.1-3.el6rhs                       @rhsqe-repo.lab.eng.blr.redhat.com_rhgs3.1-server-latest-el6_                     4.0 M

Transaction Summary
==============================================================================================================================================================================================
Remove        4 Package(s)

Installed size: 5.9 M
Is this ok [y/N]: y         
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Erasing    : glusterfs-server-3.7.1-3.el6rhs.x86_64                                                                                                                                     1/4 
  Erasing    : glusterfs-fuse-3.7.1-3.el6rhs.x86_64                                                                                                                                       2/4 
  Erasing    : glusterfs-api-3.7.1-3.el6rhs.x86_64                                                                                                                                        3/4 
  Erasing    : glusterfs-3.7.1-3.el6rhs.x86_64                                                                                                                                            4/4 
rhel-6-server-rpms/productid                                                                                                                                           | 1.7 kB     00:00     
  Verifying  : glusterfs-fuse-3.7.1-3.el6rhs.x86_64                                                                                                                                       1/4 
  Verifying  : glusterfs-api-3.7.1-3.el6rhs.x86_64                                                                                                                                        2/4 
  Verifying  : glusterfs-3.7.1-3.el6rhs.x86_64                                                                                                                                            3/4 
  Verifying  : glusterfs-server-3.7.1-3.el6rhs.x86_64                                                                                                                                     4/4 

Removed:
  glusterfs.x86_64 0:3.7.1-3.el6rhs                                                                                                                                                           

Dependency Removed:
  glusterfs-api.x86_64 0:3.7.1-3.el6rhs                        glusterfs-fuse.x86_64 0:3.7.1-3.el6rhs                        glusterfs-server.x86_64 0:3.7.1-3.el6rhs                       

Complete!
[root@dhcp43-191 ~]# yum install glusterfs-server --nogpgcheck
Loaded plugins: product-id, security, subscription-manager
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package glusterfs-server.x86_64 0:3.7.1-4.el6rhs will be installed
--> Processing Dependency: glusterfs-libs = 3.7.1-4.el6rhs for package: glusterfs-server-3.7.1-4.el6rhs.x86_64
--> Processing Dependency: glusterfs-fuse = 3.7.1-4.el6rhs for package: glusterfs-server-3.7.1-4.el6rhs.x86_64
--> Processing Dependency: glusterfs-client-xlators = 3.7.1-4.el6rhs for package: glusterfs-server-3.7.1-4.el6rhs.x86_64
--> Processing Dependency: glusterfs-cli = 3.7.1-4.el6rhs for package: glusterfs-server-3.7.1-4.el6rhs.x86_64
--> Processing Dependency: glusterfs = 3.7.1-4.el6rhs for package: glusterfs-server-3.7.1-4.el6rhs.x86_64
--> Processing Dependency: libgfapi.so.0(GFAPI_PRIVATE_3.7.0)(64bit) for package: glusterfs-server-3.7.1-4.el6rhs.x86_64
--> Processing Dependency: libgfapi.so.0(GFAPI_PRIVATE_3.4.0)(64bit) for package: glusterfs-server-3.7.1-4.el6rhs.x86_64
--> Processing Dependency: libgfapi.so.0(GFAPI_3.7.0)(64bit) for package: glusterfs-server-3.7.1-4.el6rhs.x86_64
--> Processing Dependency: libgfapi.so.0(GFAPI_3.6.0)(64bit) for package: glusterfs-server-3.7.1-4.el6rhs.x86_64
--> Processing Dependency: libgfapi.so.0(GFAPI_3.5.1)(64bit) for package: glusterfs-server-3.7.1-4.el6rhs.x86_64
--> Processing Dependency: libgfapi.so.0(GFAPI_3.4.2)(64bit) for package: glusterfs-server-3.7.1-4.el6rhs.x86_64
--> Processing Dependency: libgfapi.so.0(GFAPI_3.4.0)(64bit) for package: glusterfs-server-3.7.1-4.el6rhs.x86_64
--> Processing Dependency: libgfapi.so.0()(64bit) for package: glusterfs-server-3.7.1-4.el6rhs.x86_64
--> Running transaction check
---> Package glusterfs.x86_64 0:3.7.1-4.el6rhs will be installed
---> Package glusterfs-api.x86_64 0:3.7.1-4.el6rhs will be installed
---> Package glusterfs-cli.x86_64 0:3.7.1-3.el6rhs will be updated
---> Package glusterfs-cli.x86_64 0:3.7.1-4.el6rhs will be an update
---> Package glusterfs-client-xlators.x86_64 0:3.7.1-3.el6rhs will be updated
---> Package glusterfs-client-xlators.x86_64 0:3.7.1-4.el6rhs will be an update
---> Package glusterfs-fuse.x86_64 0:3.7.1-4.el6rhs will be installed
---> Package glusterfs-libs.x86_64 0:3.7.1-3.el6rhs will be updated
---> Package glusterfs-libs.x86_64 0:3.7.1-4.el6rhs will be an update
--> Finished Dependency Resolution

Dependencies Resolved

==============================================================================================================================================================================================
 Package                                    Arch                     Version                             Repository                                                                      Size
==============================================================================================================================================================================================
Installing:
 glusterfs-server                           x86_64                   3.7.1-4.el6rhs                      rhsqe-repo.lab.eng.blr.redhat.com_rhgs3.1-server-latest-el6_                   1.2 M
Installing for dependencies:
 glusterfs                                  x86_64                   3.7.1-4.el6rhs                      rhsqe-repo.lab.eng.blr.redhat.com_rhgs3.1-server-latest-el6_                   386 k
 glusterfs-api                              x86_64                   3.7.1-4.el6rhs                      rhsqe-repo.lab.eng.blr.redhat.com_rhgs3.1-server-latest-el6_                    51 k
 glusterfs-fuse                             x86_64                   3.7.1-4.el6rhs                      rhsqe-repo.lab.eng.blr.redhat.com_rhgs3.1-server-latest-el6_                   100 k
Updating for dependencies:
 glusterfs-cli                              x86_64                   3.7.1-4.el6rhs                      rhsqe-repo.lab.eng.blr.redhat.com_rhgs3.1-server-latest-el6_                   146 k
 glusterfs-client-xlators                   x86_64                   3.7.1-4.el6rhs                      rhsqe-repo.lab.eng.blr.redhat.com_rhgs3.1-server-latest-el6_                   898 k
 glusterfs-libs                             x86_64                   3.7.1-4.el6rhs                      rhsqe-repo.lab.eng.blr.redhat.com_rhgs3.1-server-latest-el6_                   297 k

Transaction Summary
==============================================================================================================================================================================================
Install       4 Package(s)
Upgrade       3 Package(s)

Total download size: 3.0 M
Is this ok [y/N]: y
Downloading Packages:
(1/7): glusterfs-3.7.1-4.el6rhs.x86_64.rpm                                                                                                                             | 386 kB     00:00     
(2/7): glusterfs-api-3.7.1-4.el6rhs.x86_64.rpm                                                                                                                         |  51 kB     00:00     
(3/7): glusterfs-cli-3.7.1-4.el6rhs.x86_64.rpm                                                                                                                         | 146 kB     00:00     
(4/7): glusterfs-client-xlators-3.7.1-4.el6rhs.x86_64.rpm                                                                                                              | 898 kB     00:00     
(5/7): glusterfs-fuse-3.7.1-4.el6rhs.x86_64.rpm                                                                                                                        | 100 kB     00:00     
(6/7): glusterfs-libs-3.7.1-4.el6rhs.x86_64.rpm                                                                                                                        | 297 kB     00:00     
(7/7): glusterfs-server-3.7.1-4.el6rhs.x86_64.rpm                                                                                                                      | 1.2 MB     00:00     
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                          25 MB/s | 3.0 MB     00:00     
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Updating   : glusterfs-libs-3.7.1-4.el6rhs.x86_64                                                                                                                                      1/10 
  Updating   : glusterfs-client-xlators-3.7.1-4.el6rhs.x86_64                                                                                                                            2/10 
  Installing : glusterfs-3.7.1-4.el6rhs.x86_64                                                                                                                                           3/10 
  Installing : glusterfs-fuse-3.7.1-4.el6rhs.x86_64                                                                                                                                      4/10 
  Installing : glusterfs-api-3.7.1-4.el6rhs.x86_64                                                                                                                                       5/10 
  Updating   : glusterfs-cli-3.7.1-4.el6rhs.x86_64                                                                                                                                       6/10 
  Installing : glusterfs-server-3.7.1-4.el6rhs.x86_64                                                                                                                                    7/10 
warning: /var/lib/glusterd/vols/ozone/ozone-rebalance.vol saved as /var/lib/glusterd/vols/ozone/ozone-rebalance.vol.rpmsave
warning: /var/lib/glusterd/vols/ozone/ozone.10.70.42.147.rhs-thinbrick2-ozone.vol saved as /var/lib/glusterd/vols/ozone/ozone.10.70.42.147.rhs-thinbrick2-ozone.vol.rpmsave
warning: /var/lib/glusterd/vols/ozone/trusted-ozone.tcp-fuse.vol saved as /var/lib/glusterd/vols/ozone/trusted-ozone.tcp-fuse.vol.rpmsave
warning: /var/lib/glusterd/vols/ozone/ozone.10.70.43.191.rhs-thinbrick2-ozone.vol saved as /var/lib/glusterd/vols/ozone/ozone.10.70.43.191.rhs-thinbrick2-ozone.vol.rpmsave
warning: /var/lib/glusterd/vols/ozone/ozone.10.70.42.147.rhs-thinbrick1-ozone.vol saved as /var/lib/glusterd/vols/ozone/ozone.10.70.42.147.rhs-thinbrick1-ozone.vol.rpmsave
warning: /var/lib/glusterd/vols/ozone/ozone.10.70.42.30.rhs-thinbrick2-ozone.vol saved as /var/lib/glusterd/vols/ozone/ozone.10.70.42.30.rhs-thinbrick2-ozone.vol.rpmsave
warning: /var/lib/glusterd/vols/ozone/ozone.tcp-fuse.vol saved as /var/lib/glusterd/vols/ozone/ozone.tcp-fuse.vol.rpmsave
warning: /var/lib/glusterd/vols/ozone/ozone.10.70.42.202.rhs-thinbrick1-ozone.vol saved as /var/lib/glusterd/vols/ozone/ozone.10.70.42.202.rhs-thinbrick1-ozone.vol.rpmsave
warning: /var/lib/glusterd/vols/ozone/ozone.10.70.43.191.rhs-thinbrick1-ozone.vol saved as /var/lib/glusterd/vols/ozone/ozone.10.70.43.191.rhs-thinbrick1-ozone.vol.rpmsave
warning: /var/lib/glusterd/vols/ozone/ozone.10.70.42.30.rhs-thinbrick1-ozone.vol saved as /var/lib/glusterd/vols/ozone/ozone.10.70.42.30.rhs-thinbrick1-ozone.vol.rpmsave
warning: /var/lib/glusterd/vols/ozone/ozone.10.70.42.202.rhs-thinbrick2-ozone.vol saved as /var/lib/glusterd/vols/ozone/ozone.10.70.42.202.rhs-thinbrick2-ozone.vol.rpmsave
warning: /var/lib/glusterd/vols/gluster_shared_storage/gluster_shared_storage.10.70.42.30.rhs-brick1-gss.vol saved as /var/lib/glusterd/vols/gluster_shared_storage/gluster_shared_storage.10.70.42.30.rhs-brick1-gss.vol.rpmsave
warning: /var/lib/glusterd/vols/gluster_shared_storage/gluster_shared_storage.10.70.43.191.rhs-brick1-gss.vol saved as /var/lib/glusterd/vols/gluster_shared_storage/gluster_shared_storage.10.70.43.191.rhs-brick1-gss.vol.rpmsave
warning: /var/lib/glusterd/vols/gluster_shared_storage/gluster_shared_storage-rebalance.vol saved as /var/lib/glusterd/vols/gluster_shared_storage/gluster_shared_storage-rebalance.vol.rpmsave
warning: /var/lib/glusterd/vols/gluster_shared_storage/gluster_shared_storage.10.70.42.202.rhs-brick1-gss.vol saved as /var/lib/glusterd/vols/gluster_shared_storage/gluster_shared_storage.10.70.42.202.rhs-brick1-gss.vol.rpmsave
warning: /var/lib/glusterd/vols/gluster_shared_storage/trusted-gluster_shared_storage.tcp-fuse.vol saved as /var/lib/glusterd/vols/gluster_shared_storage/trusted-gluster_shared_storage.tcp-fuse.vol.rpmsave
warning: /var/lib/glusterd/vols/gluster_shared_storage/gluster_shared_storage.tcp-fuse.vol saved as /var/lib/glusterd/vols/gluster_shared_storage/gluster_shared_storage.tcp-fuse.vol.rpmsave
  Cleanup    : glusterfs-cli-3.7.1-3.el6rhs.x86_64                                                                                                                                       8/10 
  Cleanup    : glusterfs-client-xlators-3.7.1-3.el6rhs.x86_64                                                                                                                            9/10 
  Cleanup    : glusterfs-libs-3.7.1-3.el6rhs.x86_64                                                                                                                                     10/10 
  Verifying  : glusterfs-fuse-3.7.1-4.el6rhs.x86_64                                                                                                                                      1/10 
  Verifying  : glusterfs-libs-3.7.1-4.el6rhs.x86_64                                                                                                                                      2/10 
  Verifying  : glusterfs-api-3.7.1-4.el6rhs.x86_64                                                                                                                                       3/10 
  Verifying  : glusterfs-server-3.7.1-4.el6rhs.x86_64                                                                                                                                    4/10 
  Verifying  : glusterfs-client-xlators-3.7.1-4.el6rhs.x86_64                                                                                                                            5/10 
  Verifying  : glusterfs-cli-3.7.1-4.el6rhs.x86_64                                                                                                                                       6/10 
  Verifying  : glusterfs-3.7.1-4.el6rhs.x86_64                                                                                                                                           7/10 
  Verifying  : glusterfs-client-xlators-3.7.1-3.el6rhs.x86_64                                                                                                                            8/10 
  Verifying  : glusterfs-libs-3.7.1-3.el6rhs.x86_64                                                                                                                                      9/10 
  Verifying  : glusterfs-cli-3.7.1-3.el6rhs.x86_64                                                                                                                                      10/10 

Installed:
  glusterfs-server.x86_64 0:3.7.1-4.el6rhs                                                                                                                                                    

Dependency Installed:
  glusterfs.x86_64 0:3.7.1-4.el6rhs                           glusterfs-api.x86_64 0:3.7.1-4.el6rhs                           glusterfs-fuse.x86_64 0:3.7.1-4.el6rhs                          

Dependency Updated:
  glusterfs-cli.x86_64 0:3.7.1-4.el6rhs                      glusterfs-client-xlators.x86_64 0:3.7.1-4.el6rhs                      glusterfs-libs.x86_64 0:3.7.1-4.el6rhs                     

Complete!
[root@dhcp43-191 ~]# 


###########  updation complete     ###############

[root@dhcp43-191 ~]# rpm -qa | grep glusterfs
glusterfs-libs-3.7.1-4.el6rhs.x86_64
glusterfs-api-3.7.1-4.el6rhs.x86_64
glusterfs-client-xlators-3.7.1-4.el6rhs.x86_64
glusterfs-fuse-3.7.1-4.el6rhs.x86_64
glusterfs-cli-3.7.1-4.el6rhs.x86_64
glusterfs-3.7.1-4.el6rhs.x86_64
glusterfs-server-3.7.1-4.el6rhs.x86_64
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# service glusterd start
Starting glusterd:                                         [  OK  ]
[root@dhcp43-191 ~]# gluster peer status
Number of Peers: 3

Hostname: 10.70.42.202
Uuid: 2abb8dbb-48db-4c26-97bb-1ae1e11964f1
State: Peer in Cluster (Connected)

Hostname: 10.70.42.30
Uuid: 38189182-2e48-4f9d-a342-f595389bf1bd
State: Peer in Cluster (Connected)

Hostname: 10.70.42.147
Uuid: dc3c70a2-aa43-4bf1-ba4d-1272616b69c1
State: Peer in Cluster (Connected)
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# gluster v info
 
Volume Name: gluster_shared_storage
Type: Replicate
Volume ID: ced3ec30-654b-4bf5-956b-9e99bc51d445
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 10.70.43.191:/rhs/brick1/gss
Brick2: 10.70.42.202:/rhs/brick1/gss
Brick3: 10.70.42.30:/rhs/brick1/gss
Options Reconfigured:
performance.readdir-ahead: on
 
Volume Name: ozone
Type: Distributed-Replicate
Volume ID: 9ef1ace8-505d-4d97-aa23-4296aa685f76
Status: Started
Number of Bricks: 4 x 2 = 8
Transport-type: tcp
Bricks:
Brick1: 10.70.43.191:/rhs/thinbrick1/ozone
Brick2: 10.70.42.202:/rhs/thinbrick1/ozone
Brick3: 10.70.43.191:/rhs/thinbrick2/ozone
Brick4: 10.70.42.202:/rhs/thinbrick2/ozone
Brick5: 10.70.42.30:/rhs/thinbrick1/ozone
Brick6: 10.70.42.147:/rhs/thinbrick1/ozone
Brick7: 10.70.42.30:/rhs/thinbrick2/ozone
Brick8: 10.70.42.147:/rhs/thinbrick2/ozone
Options Reconfigured:
performance.readdir-ahead: on
storage.build-pgfid: on
changelog.changelog: on
changelog.capture-del-path: on
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# gluster peer status
Number of Peers: 3

Hostname: 10.70.42.202
Uuid: 2abb8dbb-48db-4c26-97bb-1ae1e11964f1
State: Peer in Cluster (Connected)

Hostname: 10.70.42.30
Uuid: 38189182-2e48-4f9d-a342-f595389bf1bd
State: Peer in Cluster (Connected)

Hostname: 10.70.42.147
Uuid: dc3c70a2-aa43-4bf1-ba4d-1272616b69c1
State: Peer in Cluster (Connected)
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# glusterfind post sesso1 ozone
Session sesso1 with volume ozone updated
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# gluster peer status
Number of Peers: 3

Hostname: 10.70.42.202
Uuid: 2abb8dbb-48db-4c26-97bb-1ae1e11964f1
State: Peer in Cluster (Connected)

Hostname: 10.70.42.30
Uuid: 38189182-2e48-4f9d-a342-f595389bf1bd
State: Peer in Cluster (Disconnected)

Hostname: 10.70.42.147
Uuid: dc3c70a2-aa43-4bf1-ba4d-1272616b69c1
State: Peer in Cluster (Connected)
[root@dhcp43-191 ~]# 

###### got node 3 down   ################


[root@dhcp43-191 ~]# gluster peer status
Number of Peers: 3

Hostname: 10.70.42.202
Uuid: 2abb8dbb-48db-4c26-97bb-1ae1e11964f1
State: Peer in Cluster (Connected)

Hostname: 10.70.42.30
Uuid: 38189182-2e48-4f9d-a342-f595389bf1bd
State: Peer in Cluster (Disconnected)

Hostname: 10.70.42.147
Uuid: dc3c70a2-aa43-4bf1-ba4d-1272616b69c1
State: Peer in Cluster (Connected)
[root@dhcp43-191 ~]# glusterfind pre sesso1 ozone /tmp/outo1.txt 

10.70.42.30 - pre failed: ssh: connect to host 10.70.42.30 port 22: Connection timed out

10.70.42.30 - pre failed: ssh: connect to host 10.70.42.30 port 22: Connection timed out

10.70.42.30 - cleanup failed: ssh: connect to host 10.70.42.30 port 22: No route to host

10.70.42.30 - cleanup failed: ssh: connect to host 10.70.42.30 port 22: No route to host

Generated output file /tmp/outo1.txt
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# vi /tmp/outo1.txt 
[root@dhcp43-191 ~]# glusterfind post sesso1 ozone
10.70.42.30 - post failed: ssh: connect to host 10.70.42.30 port 22: Connection timed out

10.70.42.30 - post failed: ssh: connect to host 10.70.42.30 port 22: Connection timed out

Session sesso1 with volume ozone updated
[root@dhcp43-191 ~]# date
Fri Jun 19 20:19:32 IST 2015
[root@dhcp43-191 ~]# glusterfind pre sesso1 ozone /tmp/outo1.txt 
10.70.42.30 - pre failed: ssh: connect to host 10.70.42.30 port 22: No route to host

10.70.42.30 - pre failed: ssh: connect to host 10.70.42.30 port 22: No route to host

10.70.42.30 - cleanup failed: ssh: connect to host 10.70.42.30 port 22: No route to host

10.70.42.30 - cleanup failed: ssh: connect to host 10.70.42.30 port 22: No route to host

Generated output file /tmp/outo1.txt
[root@dhcp43-191 ~]# vi /tmp/outo1.txt 
[root@dhcp43-191 ~]# date
Fri Jun 19 20:26:35 IST 2015
[root@dhcp43-191 ~]# glusterfind pre sesso1 ozone /tmp/outo1.txt 
Post command is not run after last pre, use --regenerate-outfile
[root@dhcp43-191 ~]# glusterfind pre sesso1 ozone /tmp/outo1.txt --regenerate-outfile
10.70.42.30 - pre failed: ssh: connect to host 10.70.42.30 port 22: No route to host

10.70.42.30 - pre failed: ssh: connect to host 10.70.42.30 port 22: No route to host

10.70.42.30 - cleanup failed: ssh: connect to host 10.70.42.30 port 22: No route to host

10.70.42.30 - cleanup failed: ssh: connect to host 10.70.42.30 port 22: No route to host

Generated output file /tmp/outo1.txt
[root@dhcp43-191 ~]# vi /tmp/outo1.txt 
[root@dhcp43-191 ~]# cat /tmp/outo1.txt 
[root@dhcp43-191 ~]# date
Fri Jun 19 20:27:38 IST 2015
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# date
Fri Jun 19 20:28:10 IST 2015
[root@dhcp43-191 ~]# glusterfind pre sesso1 ozone /tmp/outo1.txt --regenerate-outfile
10.70.42.30 - pre failed: ssh: connect to host 10.70.42.30 port 22: No route to host

10.70.42.30 - pre failed: ssh: connect to host 10.70.42.30 port 22: No route to host

10.70.42.30 - cleanup failed: ssh: connect to host 10.70.42.30 port 22: No route to host

10.70.42.30 - cleanup failed: ssh: connect to host 10.70.42.30 port 22: No route to host

Generated output file /tmp/outo1.txt
[root@dhcp43-191 ~]# cat /tmp/outo1.txt 
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# ls -l /tmp/outo1.txt 
-rw-r--r--. 1 root root 0 Jun 19 20:28 /tmp/outo1.txt
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# glusterfind pre sesso2 ozone /tmp/outo2.txt 
10.70.42.30 - pre failed: ssh: connect to host 10.70.42.30 port 22: No route to host

10.70.42.30 - pre failed: ssh: connect to host 10.70.42.30 port 22: No route to host

10.70.42.30 - cleanup failed: ssh: connect to host 10.70.42.30 port 22: No route to host

10.70.42.30 - cleanup failed: ssh: connect to host 10.70.42.30 port 22: No route to host

Generated output file /tmp/outo2.txt
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# ls -l /tmp/outo2.txt 
-rw-r--r--. 1 root root 10233 Jun 19 20:28 /tmp/outo2.txt
[root@dhcp43-191 ~]# vi /tmp/outo2.txt 
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# glusterfind post sesso2 ozone 
10.70.42.30 - post failed: ssh: connect to host 10.70.42.30 port 22: No route to host

10.70.42.30 - post failed: ssh: connect to host 10.70.42.30 port 22: No route to host

Session sesso2 with volume ozone updated
[root@dhcp43-191 ~]# glusterfind pre sesso1 ozone /tmp/outo1.txt --regenerate-outfile
10.70.42.30 - pre failed: ssh: connect to host 10.70.42.30 port 22: No route to host

10.70.42.30 - pre failed: ssh: connect to host 10.70.42.30 port 22: No route to host

10.70.42.30 - cleanup failed: ssh: connect to host 10.70.42.30 port 22: No route to host


10.70.42.30 - cleanup failed: ssh: connect to host 10.70.42.30 port 22: Connection timed out

Generated output file /tmp/outo1.txt
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# ls -l /tmp/outo1.txt 
-rw-r--r--. 1 root root 0 Jun 19 20:34 /tmp/outo1.txt
[root@dhcp43-191 ~]#

Comment 3 Sweta Anandpara 2015-06-19 10:04:13 UTC


Client logs:

root@dhcp43-71 ~]# 
[root@dhcp43-71 ~]# mount | grep ozone
10.70.42.202:/ozone on /mnt/ozone type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072)
[root@dhcp43-71 ~]# cd /mnt/ozone
[root@dhcp43-71 ozone]# ls -a
.  ..  5582ba18%%FTZ39AW4RV.tar.gz  5582baeb%%SNRP8ENTE9  etc  level01  level20  level20_sln  .trashcan  V6MO_newhdln
[root@dhcp43-71 ozone]# 
[root@dhcp43-71 ozone]# 
[root@dhcp43-71 ozone]# tail /etc/passwd
ntp:x:38:38::/etc/ntp:/sbin/nologin
saslauth:x:499:76:Saslauthd user:/var/empty/saslauth:/sbin/nologin
postfix:x:89:89::/var/spool/postfix:/sbin/nologin
sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin
oprofile:x:16:16:Special user account to be used by OProfile:/home/oprofile:/sbin/nologin
tcpdump:x:72:72::/:/sbin/nologin
usr1:x:500:501::/home/usr1:/bin/bash
u1:x:501:500::/home/u1:/bin/bash
u2:x:502:500::/home/u2:/bin/bash
u3:x:503:0::/home/u3:/bin/bash
[root@dhcp43-71 ozone]# 
[root@dhcp43-71 ozone]# 
[root@dhcp43-71 ozone]# ls -ld level*
drwxr-xr-x. 3 root root  134 Jun 18 18:36 level01
drwxrwxrwx. 3 root g1   4203 Jun 18 21:07 level20
lrwxrwxrwx. 1 root root    7 Jun 18 21:00 level20_sln -> level20
[root@dhcp43-71 ozone]# 
[root@dhcp43-71 ozone]# 
[root@dhcp43-71 ozone]# 
[root@dhcp43-71 ozone]# chgrp g1 level01
[root@dhcp43-71 ozone]# chown u1 level01
[root@dhcp43-71 ozone]# 
[root@dhcp43-71 ozone]# ls -ld level*
drwxr-xr-x. 3 u1   g1    134 Jun 18 18:36 level01
drwxrwxrwx. 3 root g1   4203 Jun 18 21:07 level20
lrwxrwxrwx. 1 root root    7 Jun 18 21:00 level20_sln -> level20
[root@dhcp43-71 ozone]# 
[root@dhcp43-71 ozone]# chown u1 level20_sln
[root@dhcp43-71 ozone]# ls -ld level*
drwxr-xr-x. 3 u1   g1    134 Jun 18 18:36 level01
drwxrwxrwx. 3 u1   g1   4203 Jun 18 21:07 level20
lrwxrwxrwx. 1 root root    7 Jun 18 21:00 level20_sln -> level20
[root@dhcp43-71 ozone]# ls -ld level*
drwxr-xr-x. 3 u1   g1    134 Jun 18 18:36 level01
drwxrwxrwx. 3 u1   g1   4203 Jun 18 21:07 level20
lrwxrwxrwx. 1 root root    7 Jun 18 21:00 level20_sln -> level20
[root@dhcp43-71 ozone]# chown u2 level01
[root@dhcp43-71 ozone]# ls -ld level*
drwxr-xr-x. 3 u2   g1    134 Jun 18 18:36 level01
drwxrwxrwx. 3 u1   g1   4203 Jun 18 21:07 level20
lrwxrwxrwx. 1 root root    7 Jun 18 21:00 level20_sln -> level20
[root@dhcp43-71 ozone]# su u1
[u1@dhcp43-71 ozone]$ cd level01
[u1@dhcp43-71 level01]$ ls -lrt
total 204
-rw-r--r--. 1 root root 102400 Jun 18 18:36 5582c255%%4IDQ2AUHU8
-rw-r--r--. 1 root root 102400 Jun 18 18:36 5582c255%%BKCRD1RKEW
drwxr-xr-x. 3 root root   4156 Jun 18 18:36 level11
[u1@dhcp43-71 level01]$ chown u1 level11
chown: changing ownership of `level11': Operation not permitted
[u1@dhcp43-71 level01]$ chmod 777 level111
chmod: cannot access `level111': No such file or directory
[u1@dhcp43-71 level01]$ chmod 777 level11
chmod: changing permissions of `level11': Operation not permitted
[u1@dhcp43-71 level01]$ 
[u1@dhcp43-71 level01]$ 
[u1@dhcp43-71 level01]$ 
[u1@dhcp43-71 level01]$ 
[u1@dhcp43-71 level01]$ date
Fri Jun 19 15:41:07 IST 2015
[u1@dhcp43-71 level01]$ pwd
/mnt/ozone/level01
[u1@dhcp43-71 level01]$ cd ..
[u1@dhcp43-71 ozone]$ ls -a
.  ..  5582ba18%%FTZ39AW4RV.tar.gz  5582baeb%%SNRP8ENTE9  etc  level01  level20  level20_sln  .trashcan  V6MO_newhdln
[u1@dhcp43-71 ozone]$ 
[u1@dhcp43-71 ozone]$ 
[u1@dhcp43-71 ozone]$ touch level20_sln/
[u1@dhcp43-71 ozone]$ ls -ld level*
drwxr-xr-x. 3 u2   g1    134 Jun 18 18:36 level01
drwxrwxrwx. 3 u1   g1   4203 Jun 19 15:41 level20
lrwxrwxrwx. 1 root root    7 Jun 18 21:00 level20_sln -> level20
[u1@dhcp43-71 ozone]$ 
[u1@dhcp43-71 ozone]$ 
[u1@dhcp43-71 ozone]$ mv level20_sln/ level20_sln2
mv: cannot move `level20_sln/' to `level20_sln2': Not a directory
[u1@dhcp43-71 ozone]$ mv level20_sln level20_sln2
mv: cannot move `level20_sln' to `level20_sln2': Permission denied
[u1@dhcp43-71 ozone]$ ls -lrt
total 321
drwxr-xr-x. 100 root root  16384 Jun 18 17:51 etc
-rw-r--r--.   1 root root 102728 Jun 18 18:07 5582ba18%%FTZ39AW4RV.tar.gz
drwxr-xr-x.   3 u2   g1      134 Jun 18 18:36 level01
-rwxr-xr-x.   2 u1   g1   102400 Jun 18 18:36 V6MO_newhdln
-rw-r--r--.   1 root root 102400 Jun 18 20:47 5582baeb%%SNRP8ENTE9
lrwxrwxrwx.   1 root root      7 Jun 18 21:00 level20_sln -> level20
drwxrwxrwx.   3 u1   g1     4203 Jun 19 15:41 level20
[u1@dhcp43-71 ozone]$ exit
[root@dhcp43-71 ozone]# 
[root@dhcp43-71 ozone]# 
[root@dhcp43-71 ozone]# mv level20_sln level20_sln2
[root@dhcp43-71 ozone]# ls -lrt
total 321
drwxr-xr-x. 100 root root  16384 Jun 18 17:51 etc
-rw-r--r--.   1 root root 102728 Jun 18 18:07 5582ba18%%FTZ39AW4RV.tar.gz
drwxr-xr-x.   3 u2   g1      134 Jun 18 18:36 level01
-rwxr-xr-x.   2 u1   g1   102400 Jun 18 18:36 V6MO_newhdln
-rw-r--r--.   1 root root 102400 Jun 18 20:47 5582baeb%%SNRP8ENTE9
lrwxrwxrwx.   1 root root      7 Jun 18 21:00 level20_sln2 -> level20
drwxrwxrwx.   3 u1   g1     4203 Jun 19 15:41 level20
[root@dhcp43-71 ozone]# 
[root@dhcp43-71 ozone]# 
[root@dhcp43-71 ozone]# ln -s level01 level20_sln2
[root@dhcp43-71 ozone]# ls -lrt
total 325
drwxr-xr-x. 100 root root  16384 Jun 18 17:51 etc
-rw-r--r--.   1 root root 102728 Jun 18 18:07 5582ba18%%FTZ39AW4RV.tar.gz
drwxr-xr-x.   3 u2   g1      134 Jun 18 18:36 level01
-rwxr-xr-x.   2 u1   g1   102400 Jun 18 18:36 V6MO_newhdln
-rw-r--r--.   1 root root 102400 Jun 18 20:47 5582baeb%%SNRP8ENTE9
lrwxrwxrwx.   1 root root      7 Jun 18 21:00 level20_sln2 -> level20
drwxrwxrwx.   3 u1   g1     8252 Jun 19 15:44 level20
[root@dhcp43-71 ozone]# 
[root@dhcp43-71 ozone]# 
[root@dhcp43-71 ozone]# ls -lrt
total 325
drwxr-xr-x. 100 root root  16384 Jun 18 17:51 etc
-rw-r--r--.   1 root root 102728 Jun 18 18:07 5582ba18%%FTZ39AW4RV.tar.gz
drwxr-xr-x.   3 u2   g1      134 Jun 18 18:36 level01
-rwxr-xr-x.   2 u1   g1   102400 Jun 18 18:36 V6MO_newhdln
-rw-r--r--.   1 root root 102400 Jun 18 20:47 5582baeb%%SNRP8ENTE9
lrwxrwxrwx.   1 root root      7 Jun 18 21:00 level20_sln2 -> level20
drwxrwxrwx.   3 u1   g1     8252 Jun 19 15:44 level20
[root@dhcp43-71 ozone]# ln -s level01 level20_sln2
ln: creating symbolic link `level20_sln2/level01': File exists
[root@dhcp43-71 ozone]# cd level20_sln2/
5582c254%%HVP139N6JI  5582c254%%UF1W41V6M0  level01               level30/              V6MO_hdln_sln         
[root@dhcp43-71 ozone]# cd level20_sln2/^C
[root@dhcp43-71 ozone]# 
[root@dhcp43-71 ozone]# 
[root@dhcp43-71 ozone]# 
[root@dhcp43-71 ozone]# touch a
[root@dhcp43-71 ozone]# ln -s a a_symlink
[root@dhcp43-71 ozone]# 
[root@dhcp43-71 ozone]# 
[root@dhcp43-71 ozone]# ls -lrt
total 326
drwxr-xr-x. 100 root root  16384 Jun 18 17:51 etc
-rw-r--r--.   1 root root 102728 Jun 18 18:07 5582ba18%%FTZ39AW4RV.tar.gz
drwxr-xr-x.   3 u2   g1      134 Jun 18 18:36 level01
-rwxr-xr-x.   2 u1   g1   102400 Jun 18 18:36 V6MO_newhdln
-rw-r--r--.   1 root root 102400 Jun 18 20:47 5582baeb%%SNRP8ENTE9
lrwxrwxrwx.   1 root root      7 Jun 18 21:00 level20_sln2 -> level20
drwxrwxrwx.   3 u1   g1     8252 Jun 19 15:44 level20
-rw-r--r--.   1 root root      0 Jun 19 15:46 a
lrwxrwxrwx.   1 root root      1 Jun 19 15:46 a_symlink -> a
[root@dhcp43-71 ozone]# 
[root@dhcp43-71 ozone]# 
[root@dhcp43-71 ozone]# touch b
[root@dhcp43-71 ozone]# ln -s b a_symlink_to_b
[root@dhcp43-71 ozone]# ls -lrt
total 326
drwxr-xr-x. 100 root root  16384 Jun 18 17:51 etc
-rw-r--r--.   1 root root 102728 Jun 18 18:07 5582ba18%%FTZ39AW4RV.tar.gz
drwxr-xr-x.   3 u2   g1      134 Jun 18 18:36 level01
-rwxr-xr-x.   2 u1   g1   102400 Jun 18 18:36 V6MO_newhdln
-rw-r--r--.   1 root root 102400 Jun 18 20:47 5582baeb%%SNRP8ENTE9
lrwxrwxrwx.   1 root root      7 Jun 18 21:00 level20_sln2 -> level20
drwxrwxrwx.   3 u1   g1     8252 Jun 19 15:44 level20
-rw-r--r--.   1 root root      0 Jun 19 15:46 a
lrwxrwxrwx.   1 root root      1 Jun 19 15:46 a_symlink -> a
-rw-r--r--.   1 root root      0 Jun 19 15:47 b
lrwxrwxrwx.   1 root root      1 Jun 19 15:47 a_symlink_to_b -> b
[root@dhcp43-71 ozone]# 
[root@dhcp43-71 ozone]# 
[root@dhcp43-71 ozone]# ln -s b a_symlink
ln: creating symbolic link `a_symlink': File exists
[root@dhcp43-71 ozone]# rm -f a_symlink*
[root@dhcp43-71 ozone]# ls -lrt
total 325
drwxr-xr-x. 100 root root  16384 Jun 18 17:51 etc
-rw-r--r--.   1 root root 102728 Jun 18 18:07 5582ba18%%FTZ39AW4RV.tar.gz
drwxr-xr-x.   3 u2   g1      134 Jun 18 18:36 level01
-rwxr-xr-x.   2 u1   g1   102400 Jun 18 18:36 V6MO_newhdln
-rw-r--r--.   1 root root 102400 Jun 18 20:47 5582baeb%%SNRP8ENTE9
lrwxrwxrwx.   1 root root      7 Jun 18 21:00 level20_sln2 -> level20
drwxrwxrwx.   3 u1   g1     8252 Jun 19 15:44 level20
-rw-r--r--.   1 root root      0 Jun 19 15:46 a
-rw-r--r--.   1 root root      0 Jun 19 15:47 b
[root@dhcp43-71 ozone]# 
[root@dhcp43-71 ozone]# 
[root@dhcp43-71 ozone]# cd level01
[root@dhcp43-71 level01]# ls -lrt
total 204
-rw-r--r--. 1 root root 102400 Jun 18 18:36 5582c255%%4IDQ2AUHU8
-rw-r--r--. 1 root root 102400 Jun 18 18:36 5582c255%%BKCRD1RKEW
drwxr-xr-x. 3 root root   4156 Jun 18 18:36 level11
[root@dhcp43-71 level01]# cd level11/
[root@dhcp43-71 level11]# ls -lrt
total 200
-rw-r--r--. 1 root root 102400 Jun 18 18:36 5582c255%%MSNPHQ27AD
-rw-r--r--. 1 root root 102400 Jun 18 18:36 5582c255%%FLGHY1NZ1A
drwxr-xr-x. 3 root root    134 Jun 18 18:36 level21
[root@dhcp43-71 level11]# cd level21/
5582c255%%DMGID3IO9G  5582c256%%F8GH8RXVKJ  level31/              
[root@dhcp43-71 level11]# cd level21/
[root@dhcp43-71 level21]# ls -lrt
total 208
-rw-r--r--. 1 root root 102400 Jun 18 18:36 5582c255%%DMGID3IO9G
-rw-r--r--. 1 root root 102400 Jun 18 18:36 5582c256%%F8GH8RXVKJ
drwxr-xr-x. 2 root root   8204 Jun 18 18:51 level31
[root@dhcp43-71 level21]# man chgrp
[root@dhcp43-71 level21]# ls -lrt level31/
total 400
-rw-r--r--. 1 root root 102400 Jun 18 18:36 5582c255%%OQ65TSMRUD
-rw-r--r--. 1 root root 102400 Jun 18 18:36 5582c255%%WRBYSGT7PZ
-rw-r--r--. 1 root root 102400 Jun 18 18:36 5582c256%%1QNX332SKS
-rw-r--r--. 1 root root 102400 Jun 18 18:36 5582c256%%MPIBDVQJQ4
[root@dhcp43-71 level21]# chgrp -R g1 level31
[root@dhcp43-71 level21]# ls -lrt level31/
total 400
-rw-r--r--. 1 root g1 102400 Jun 18 18:36 5582c255%%OQ65TSMRUD
-rw-r--r--. 1 root g1 102400 Jun 18 18:36 5582c255%%WRBYSGT7PZ
-rw-r--r--. 1 root g1 102400 Jun 18 18:36 5582c256%%1QNX332SKS
-rw-r--r--. 1 root g1 102400 Jun 18 18:36 5582c256%%MPIBDVQJQ4
[root@dhcp43-71 level21]# ls -ld level31/
drwxr-xr-x. 2 root g1 8204 Jun 18 18:51 level31/
[root@dhcp43-71 level21]# cd ../../
[root@dhcp43-71 level01]# ls -lrt
total 204
-rw-r--r--. 1 root root 102400 Jun 18 18:36 5582c255%%4IDQ2AUHU8
-rw-r--r--. 1 root root 102400 Jun 18 18:36 5582c255%%BKCRD1RKEW
drwxr-xr-x. 3 root root   4156 Jun 18 18:36 level11
[root@dhcp43-71 level01]# pwd
/mnt/ozone/level01
[root@dhcp43-71 level01]# cd ..
[root@dhcp43-71 ozone]# 
[root@dhcp43-71 ozone]# 
[root@dhcp43-71 ozone]# cd level11
-bash: cd: level11: No such file or directory
[root@dhcp43-71 ozone]# cd level
level01/      level20/      level20_sln2/ 
[root@dhcp43-71 ozone]# cd level01/level11/
[root@dhcp43-71 level11]# 
[root@dhcp43-71 level11]# 
[root@dhcp43-71 level11]# ls -lrt
total 200
-rw-r--r--. 1 root root 102400 Jun 18 18:36 5582c255%%MSNPHQ27AD
-rw-r--r--. 1 root root 102400 Jun 18 18:36 5582c255%%FLGHY1NZ1A
drwxr-xr-x. 3 root root    134 Jun 18 18:36 level21
[root@dhcp43-71 level11]# 
[root@dhcp43-71 level11]# 
[root@dhcp43-71 level11]# ln hardlink_file 5582c255%%MSNPHQ27AD
ln: accessing `hardlink_file': No such file or directory
[root@dhcp43-71 level11]# ln 5582c255%%MSNPHQ27AD hardlink_file
[root@dhcp43-71 level11]# ls -lrt
total 300
-rw-r--r--. 2 root root 102400 Jun 18 18:36 hardlink_file
-rw-r--r--. 2 root root 102400 Jun 18 18:36 5582c255%%MSNPHQ27AD
-rw-r--r--. 1 root root 102400 Jun 18 18:36 5582c255%%FLGHY1NZ1A
drwxr-xr-x. 3 root root    134 Jun 18 18:36 level21
[root@dhcp43-71 level11]# 
[root@dhcp43-71 level11]# 
[root@dhcp43-71 level11]# mv hardlink_file ../hardlink
[root@dhcp43-71 level11]# ls -lrt
total 200
-rw-r--r--. 2 root root 102400 Jun 18 18:36 5582c255%%MSNPHQ27AD
-rw-r--r--. 1 root root 102400 Jun 18 18:36 5582c255%%FLGHY1NZ1A
drwxr-xr-x. 3 root root    134 Jun 18 18:36 level21
[root@dhcp43-71 level11]# ls -lrt ../
total 304
-rw-r--r--. 1 root root 102400 Jun 18 18:36 5582c255%%4IDQ2AUHU8
-rw-r--r--. 1 root root 102400 Jun 18 18:36 5582c255%%BKCRD1RKEW
-rw-r--r--. 2 root root 102400 Jun 18 18:36 hardlink
drwxr-xr-x. 3 root root   4156 Jun 19 15:54 level11
[root@dhcp43-71 level11]# cd
[root@dhcp43-71 ~]# 
[root@dhcp43-71 ~]# 
[root@dhcp43-71 ~]# 
[root@dhcp43-71 ~]# mount | grep ozone
10.70.42.202:/ozone on /mnt/ozone type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072)
[root@dhcp43-71 ~]# 
[root@dhcp43-71 ~]# 
[root@dhcp43-71 ~]# 
[root@dhcp43-71 ~]# 
[root@dhcp43-71 ~]# 
[root@dhcp43-71 ~]# 
[root@dhcp43-71 ~]# cd /mnt/ozone
[root@dhcp43-71 ozone]# 
[root@dhcp43-71 ozone]# 
[root@dhcp43-71 ozone]# df -k .
Filesystem          1K-blocks   Used Available Use% Mounted on
10.70.42.202:/ozone  33511424 172672  33338752   1% /mnt/ozone
[root@dhcp43-71 ozone]# 
[root@dhcp43-71 ozone]# 
[root@dhcp43-71 ozone]# ls -a
.  ..  5582ba18%%FTZ39AW4RV.tar.gz  5582baeb%%SNRP8ENTE9  a  b  etc  level01  level20  level20_sln2  .trashcan  V6MO_newhdln
[root@dhcp43-71 ozone]# 
[root@dhcp43-71 ozone]# 
[root@dhcp43-71 ozone]# 
[root@dhcp43-71 ozone]# crefi
usage: crefi [-h] [-V] [-n FILES] [--size SIZE] [--random] [--max MAX]
             [--min MIN] [--single] [--multi] [-b BRDTH] [-d DEPTH] [-l FLEN]
             [-t {text,sparse,binary,tar}] [-I INTER]
             [--fop {create,rename,chmod,chown,chgrp,symlink,hardlink,truncate,setxattr}]
             [-R] [-T THREADS]
             mntpnt
crefi: error: too few arguments
[root@dhcp43-71 ozone]# rm -rf level20
[root@dhcp43-71 ozone]# crefi --multi -n 2 -b 4 -d 10 -t tar .
1
2015-06-19 18:04:41,246 - thread-0: Entering the directory level00
2015-06-19 18:04:41,560 - thread-0: Entering the directory level10
2015-06-19 18:04:41,879 - thread-0: Entering the directory level20
2015-06-19 18:04:42,185 - thread-0: Entering the directory level30
2015-06-19 18:04:42,500 - thread-0: Entering the directory level40
2015-06-19 18:04:42,847 - thread-0: Entering the directory level50
2015-06-19 18:04:43,151 - thread-0: Entering the directory level60
2015-06-19 18:04:43,479 - thread-0: Entering the directory level70
2015-06-19 18:04:43,837 - thread-0: Entering the directory level80
2015-06-19 18:04:44,168 - thread-0: Entering the directory level90
2015-06-19 18:04:44,463 - thread-0: Entering the directory level01
2015-06-19 18:04:44,714 - thread-0: Entering the directory level11
2015-06-19 18:04:44,967 - thread-0: Entering the directory level21
2015-06-19 18:04:45,212 - thread-0: Entering the directory level31
2015-06-19 18:04:45,520 - thread-0: Entering the directory level41
2015-06-19 18:04:45,863 - thread-0: Entering the directory level51
2015-06-19 18:04:46,186 - thread-0: Entering the directory level61
2015-06-19 18:04:46,551 - thread-0: Entering the directory level71
2015-06-19 18:04:46,909 - thread-0: Entering the directory level81
2015-06-19 18:04:47,223 - thread-0: Entering the directory level91
2015-06-19 18:04:47,549 - thread-0: Entering the directory level02
2015-06-19 18:04:47,832 - thread-0: Entering the directory level12
2015-06-19 18:04:48,134 - thread-0: Entering the directory level22
2015-06-19 18:04:48,437 - thread-0: Entering the directory level32
2015-06-19 18:04:48,751 - thread-0: Entering the directory level42
2015-06-19 18:04:49,064 - thread-0: Entering the directory level52
2015-06-19 18:04:49,385 - thread-0: Entering the directory level62
2015-06-19 18:04:49,695 - thread-0: Entering the directory level72
2015-06-19 18:04:49,999 - thread-0: Entering the directory level82
2015-06-19 18:04:50,306 - thread-0: Entering the directory level92
2015-06-19 18:04:50,664 - thread-0: Entering the directory level03
2015-06-19 18:04:50,955 - thread-0: Entering the directory level13
2015-06-19 18:04:51,311 - thread-0: Entering the directory level23
2015-06-19 18:04:51,628 - thread-0: Entering the directory level33
2015-06-19 18:04:51,950 - thread-0: Entering the directory level43
2015-06-19 18:04:52,262 - thread-0: Entering the directory level53
2015-06-19 18:04:52,597 - thread-0: Entering the directory level63
2015-06-19 18:04:52,913 - thread-0: Entering the directory level73
2015-06-19 18:04:53,227 - thread-0: Entering the directory level83
2015-06-19 18:04:53,542 - thread-0: Entering the directory level93
2015-06-19 18:04:53,872 - thread-0: finished creating files with throughput ---- 2.598 MBps
[root@dhcp43-71 ozone]# 
[root@dhcp43-71 ozone]# 
[root@dhcp43-71 ozone]# 
[root@dhcp43-71 ozone]# ls -a
.  ..  5582ba18%%FTZ39AW4RV.tar.gz  5582baeb%%SNRP8ENTE9  a  b  etc  level00  level01  level02  level03  level20_sln2  .trashcan  V6MO_newhdln
[root@dhcp43-71 ozone]# 
[root@dhcp43-71 ozone]# 
[root@dhcp43-71 ozone]# cd level01/
[root@dhcp43-71 level01]# ls
5582c255%%4IDQ2AUHU8  5582c255%%BKCRD1RKEW  55840c64%%KDYOF9Y4XD.tar.gz  55840c64%%MPVF41SC4I.tar.gz  hardlink  level11
[root@dhcp43-71 level01]# ls -lrt level11/
total 406
-rw-r--r--. 2 root root 102400 Jun 18 18:36 5582c255%%MSNPHQ27AD
-rw-r--r--. 1 root root 102400 Jun 18 18:36 5582c255%%FLGHY1NZ1A
-rw-r--r--. 1 root root 102748 Jun 19 18:04 55840c64%%H4NYV4N8FY.tar.gz
-rw-r--r--. 1 root root 102750 Jun 19 18:04 55840c64%%EWWTBU4WD1.tar.gz
drwxr-xr-x. 3 root root   4217 Jun 19 18:04 level21
[root@dhcp43-71 level01]# ls -lrt level11/level21/
total 406
-rw-r--r--. 1 root root 102400 Jun 18 18:36 5582c256%%F8GH8RXVKJ
-rw-r--r--. 1 root root 102400 Jun 18 18:36 5582c255%%DMGID3IO9G
-rw-r--r--. 1 root root 102762 Jun 19 18:04 55840c64%%JAS3FVJCJF.tar.gz
drwxr-xr-x. 3 root g1     4237 Jun 19 18:04 level31
-rw-r--r--. 1 root root 102770 Jun 19 18:04 55840c65%%PZML6HA4Y2.tar.gz
[root@dhcp43-71 level01]# ls -lrt level11/level21/level31/
total 603
-rw-r--r--. 1 root g1   102400 Jun 18 18:36 5582c255%%OQ65TSMRUD
-rw-r--r--. 1 root g1   102400 Jun 18 18:36 5582c255%%WRBYSGT7PZ
-rw-r--r--. 1 root g1   102400 Jun 18 18:36 5582c256%%MPIBDVQJQ4
-rw-r--r--. 1 root g1   102400 Jun 18 18:36 5582c256%%1QNX332SKS
-rw-r--r--. 1 root root 102773 Jun 19 18:04 55840c65%%H68PIMOPLI.tar.gz
-rw-r--r--. 1 root root 102775 Jun 19 18:04 55840c65%%B7UQFKODEC.tar.gz
drwxr-xr-x. 3 root root    148 Jun 19 18:04 level41
[root@dhcp43-71 level01]# ls -lrt level11/level21/level31/level41/
total 203
-rw-r--r--. 1 root root 102785 Jun 19 18:04 55840c65%%B54R31P2H6.tar.gz
drwxr-xr-x. 3 root root    148 Jun 19 18:04 level51
-rw-r--r--. 1 root root 102779 Jun 19 18:04 55840c65%%KY7890UJVK.tar.gz
[root@dhcp43-71 level01]# ls -lrt level11/level21/level31/level41/level51/
total 203
-rw-r--r--. 1 root root 102799 Jun 19 18:04 55840c65%%5PR73SKQMG.tar.gz
-rw-r--r--. 1 root root 102794 Jun 19 18:04 55840c66%%V67OV1DJZY.tar.gz
drwxr-xr-x. 3 root root    148 Jun 19 18:04 level61
[root@dhcp43-71 level01]# ^C
[root@dhcp43-71 level01]# 
[root@dhcp43-71 level01]# 
[root@dhcp43-71 level01]# mv level11/level21/level31/level41/level51/level61/level71/level81/level91/55840c67%%
55840c67%%SDS87LVZNA.tar.gz  55840c67%%ZBNTKUGCSO.tar.gz  
[root@dhcp43-71 level01]# mv level11/level21/level31/level41/level51/level61/level71/level81/level91/55840c67%%
55840c67%%SDS87LVZNA.tar.gz  55840c67%%ZBNTKUGCSO.tar.gz  
[root@dhcp43-71 level01]# mv level11/level21/level31/level41/level51/level61/level71/level81/level91/55840c67%%ZBNTKUGCSO.tar.gz  55840c67%%ZBNTKUGCSZZZZZZZ.tar.gz
[root@dhcp43-71 level01]# ls
5582c255%%4IDQ2AUHU8  5582c255%%BKCRD1RKEW  55840c64%%KDYOF9Y4XD.tar.gz  55840c64%%MPVF41SC4I.tar.gz  55840c67%%ZBNTKUGCSZZZZZZZ.tar.gz  hardlink  level11
[root@dhcp43-71 level01]# 
[root@dhcp43-71 level01]# 
[root@dhcp43-71 level01]# 
[root@dhcp43-71 level01]# cd ..
[root@dhcp43-71 ozone]# ls -a
.  ..  5582ba18%%FTZ39AW4RV.tar.gz  5582baeb%%SNRP8ENTE9  a  b  etc  level00  level01  level02  level03  level20_sln2  .trashcan  V6MO_newhdln
[root@dhcp43-71 ozone]# 
[root@dhcp43-71 ozone]# 
[root@dhcp43-71 ozone]# mv level00/level10/ .
[root@dhcp43-71 ozone]# ls -a
.  ..  5582ba18%%FTZ39AW4RV.tar.gz  5582baeb%%SNRP8ENTE9  a  b  etc  level00  level01  level02  level03  level10  level20_sln2  .trashcan  V6MO_newhdln
[root@dhcp43-71 ozone]# ls -lrt level10
total 203
-rw-r--r--. 1 root root 102747 Jun 19 18:04 55840c61%%1FBXTH35MS.tar.gz
-rw-r--r--. 1 root root 102748 Jun 19 18:04 55840c61%%SZRHLM6VL6.tar.gz
drwxr-xr-x. 3 root root    148 Jun 19 18:04 level20
[root@dhcp43-71 ozone]# find level10/ | wc -l
27
[root@dhcp43-71 ozone]# find level03 | wc -l
30
[root@dhcp43-71 ozone]# 
[root@dhcp43-71 ozone]# 
[root@dhcp43-71 ozone]#

Comment 4 Sweta Anandpara 2015-06-19 10:17:14 UTC

The only difference that I see between 'sesso1' and 'sesso2' is that I ran a 'glusterfind post' for 'sesso1' after updating the build, whereas it was already done for 'sesso2' before itself. Not really sure how that would affect.

Comment 5 Sweta Anandpara 2015-06-19 10:20:14 UTC

Got the node 'node3' back up and waiting for the cluster to get back to healthy state. 

Ran 'glusterfind pre' command again on 'sesso1' and it does succeed without any error (as expected- since all the nodes are up). BUT, the output file that it generates is incorrect (displaying just 4 entries, whereas >100 entries are expected)

[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# # after getting the node back up
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# gluster peer status
Number of Peers: 3

Hostname: 10.70.42.202
Uuid: 2abb8dbb-48db-4c26-97bb-1ae1e11964f1
State: Peer in Cluster (Connected)

Hostname: 10.70.42.30
Uuid: 38189182-2e48-4f9d-a342-f595389bf1bd
State: Peer in Cluster (Disconnected)

Hostname: 10.70.42.147
Uuid: dc3c70a2-aa43-4bf1-ba4d-1272616b69c1
State: Peer in Cluster (Connected)
[root@dhcp43-191 ~]#
[root@dhcp43-191 ~]# gluster peer status
Number of Peers: 3

Hostname: 10.70.42.202
Uuid: 2abb8dbb-48db-4c26-97bb-1ae1e11964f1
State: Peer in Cluster (Connected)

Hostname: 10.70.42.30
Uuid: 38189182-2e48-4f9d-a342-f595389bf1bd
State: Peer in Cluster (Connected)

Hostname: 10.70.42.147
Uuid: dc3c70a2-aa43-4bf1-ba4d-1272616b69c1
State: Peer in Cluster (Connected)
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# glusterfind pre sesso1 ozone /tmp/outo1.txt --regenerate-outfile
Generated output file /tmp/outo1.txt
[root@dhcp43-191 ~]# ls -l /tmp/outo1.txt 
-rw-r--r--. 1 root root 68 Jun 19 21:18 /tmp/outo1.txt
[root@dhcp43-191 ~]# vi /tmp/outo1.txt 
[root@dhcp43-191 ~]# ^C
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# cat /tmp/outo1.txt 
DELETE level20 
NEW level00 
NEW level02 
NEW level03 
NEW level10 
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# glusterfind pre sesso1 ozone /tmp/outo1.txt --regenerate-outfile
Generated output file /tmp/outo1.txt
[root@dhcp43-191 ~]# ls -l /tmp/outo1.txt 
-rw-r--r--. 1 root root 68 Jun 19 21:19 /tmp/outo1.txt
[root@dhcp43-191 ~]# cat /tmp/outo1.txt 
DELETE level20 
NEW level00 
NEW level02 
NEW level03 
NEW level10 
[root@dhcp43-191 ~]# gluster v status ozone
Status of volume: ozone
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.43.191:/rhs/thinbrick1/ozone    49153     0          Y       6807 
Brick 10.70.42.202:/rhs/thinbrick1/ozone    49153     0          Y       31482
Brick 10.70.43.191:/rhs/thinbrick2/ozone    49154     0          Y       6815 
Brick 10.70.42.202:/rhs/thinbrick2/ozone    49154     0          Y       31489
Brick 10.70.42.30:/rhs/thinbrick1/ozone     49153     0          Y       1999 
Brick 10.70.42.147:/rhs/thinbrick1/ozone    49152     0          Y       31451
Brick 10.70.42.30:/rhs/thinbrick2/ozone     49154     0          Y       1998 
Brick 10.70.42.147:/rhs/thinbrick2/ozone    49153     0          Y       31456
NFS Server on localhost                     2049      0          Y       6783 
Self-heal Daemon on localhost               N/A       N/A        Y       6792 
NFS Server on 10.70.42.202                  2049      0          Y       31460
Self-heal Daemon on 10.70.42.202            N/A       N/A        Y       31467
NFS Server on 10.70.42.147                  2049      0          Y       31437
Self-heal Daemon on 10.70.42.147            N/A       N/A        Y       31444
NFS Server on 10.70.42.30                   2049      0          Y       2572 
Self-heal Daemon on 10.70.42.30             N/A       N/A        Y       2580 
 
Task Status of Volume ozone
------------------------------------------------------------------------------
There are no active volume tasks
 
[root@dhcp43-191 ~]# gluster v info ozone
 
Volume Name: ozone
Type: Distributed-Replicate
Volume ID: 9ef1ace8-505d-4d97-aa23-4296aa685f76
Status: Started
Number of Bricks: 4 x 2 = 8
Transport-type: tcp
Bricks:
Brick1: 10.70.43.191:/rhs/thinbrick1/ozone
Brick2: 10.70.42.202:/rhs/thinbrick1/ozone
Brick3: 10.70.43.191:/rhs/thinbrick2/ozone
Brick4: 10.70.42.202:/rhs/thinbrick2/ozone
Brick5: 10.70.42.30:/rhs/thinbrick1/ozone
Brick6: 10.70.42.147:/rhs/thinbrick1/ozone
Brick7: 10.70.42.30:/rhs/thinbrick2/ozone
Brick8: 10.70.42.147:/rhs/thinbrick2/ozone
Options Reconfigured:
performance.readdir-ahead: on
storage.build-pgfid: on
changelog.changelog: on
changelog.capture-del-path: on
[root@dhcp43-191 ~]#

Comment 6 Sweta Anandpara 2015-06-23 04:41:15 UTC

Sosreports updated at: http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/1233622/

Comment 8 Amar Tumballi 2018-04-16 03:04:09 UTC

Feel free to open this bug if the issue still persists and you require a fix. Closing this as WONTFIX as we are not working on this bug, and treating it as a 'TIMEOUT'.