Bug 782249

Description of problem:

There is a problem when renaming files.  The flow of the application is below:

=================
GET LOCK

Open file ‘log.last’

If file size > MAX then
   close file
   rename  log.last to  log.<n>
   create and open  ‘log.last’

Write record to file.
Close file

UNLOCK
=================

What I see is that sometimes after the rename, a process will open the ‘last.log’ file but the file they will actually have opened if the renamed file.



Version-Release number of selected component (if applicable): 3.2.5

To reproduce the issue, I'm going to attach 3 files.  1 is the test script that reproduces the issue, one is the sample data I run the test script on, and 1 is the Gluster logs at the time of the issue:

I have attached 3 files:

glusterfs.tar.gz: Contains the gluster log created during the test.
Test_src.tar.gz: Contains the source code for the test application.
Td_test.tar.gz: Contains the test environment and test log files from the last test.

This is the gluster volume info:
===================
[root@localhost ~]# gluster volume info all

Volume Name: glustervol
Type: Distributed-Replicate
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 192.168.3.13:/xfs1
Brick2: 192.168.3.14:/xfs1
Brick3: 192.168.3.15:/xfs1
Brick4: 192.168.3.16:/xfs1
Options Reconfigured:
performance.flush-behind: On
geo-replication.indexing: Off
performance.cache-refresh-timeout: 1
performance.read-ahead: on
performance.quick-read: off
[root@localhost ~]# 
==================================


============================
If you look in the logs folder in the test environment you will see that log.1 and  log.2 are identical which indicates that the ‘last_log’ was renamed twice.


If you do a dump of the logs (./TDLogTest T logs 1) you will also see the last records:
 194 ( 1018677) {DEADBEAF,  1018677,  4,      6,   4092}
 195 ( 1022805) {DEADBEAF,  1022805,  5,      6,   1240}
 196 ( 1024081) {0,  1024081,  3,      0,      0}
 197 ( 1024117) {0,  1024081,  6,      0,      0}

The last 2 records are log terminating records of which there should only be one.



The test code is fairly straight forward. You will see that I use flock() on the file logs/logInfo to control access to updating of the logs.
(There are a few things in the logs folder that are there from older versions of the test and you will also find code in the test which is no longer used.)




=======================
To perform the test:
Copy the test environment into the gluster volume.

Before starting the test one machine needs to reset the test environment: 
./resetTDLogTest

To run the test each machine involved in the test executes the script:
./startTDLogTest

This will start 3 test writers and 1 test reader in the background.
A test clients will stop if an error is detected.
To manually stop the test execute the scrip:
./stopTDLogTest
On one of the test machines.

You can dump the records in a log with the command:
./TDLogTest T logs <log number>