Description of problem: * log rotate eg. with rotate count as 2 and with the following files created x.log, x.log.1 and x.log.2, another iteration of logrotate results in the following operations to be performed x.log.2 is renamed to x.log.3 x.log.1 is renamed to x.log.2 x.log is renamed to x.log.1 x.log.3 is deleted x.log is created [possible gfid allocated that belonged to x.log.3] when a file is created, there's a big possibility that a gfid used earlier can be reassigned and reused. This causes RENAME operations to fail on slave nodes when logs are replayed, typically on a Georep restart. A function called logrotate_simulate has been added to tests/geo-rep.rc to help simulate this situation. Starting from a clean state, logrotate_simulate has to be called 4 times with a rotate count of 2 to help the logs roll over and cause a delete at the end and a gfid reallocation. With the bug-fix in place, this test should not cause the Georep session to go into a Faulty state. * create+rename On log replay after Georep restart, a create+rename causes an entry to be created for the original file, but it cannot be linked to the gfid back-end since it is associated with the renamed file before the Georep restart. A function create_rename simulates the create+rename scenario and the function create_rename_ok tests if the dangling entry is present at the slave mount. With the bug-fix in place, a dangling entry with the original name should not be found at the slave mount after logs are replayed after a Georep restart. * hard-link rename This case is similar to the create+rename case except that this is a case of renaming hard-link to one of its other names. A function hardlink_rename has been added to tests/geo-rep.rc which simulates the creation and rename of hard-link. After a Georep session restart, the test function hardlink_rename_ok should not find the source link name on the slave. With the bug-fix in place, this test should not fail. If changelogs have been enabled on the slave as well, then the logs should show an UNLINK entry for the source link name, since a rename operation of a hard link to one of its names essentially just drops the source name as per the 'mv' command semantics. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
"tests" component is for tests framework only. File a bug under test component if you find a bug in 1. any of the *.rc files under tests/ 2. run-tests.sh For everything else, the bug should be filed on 1. component which is being tested by .t file if the .t file requires fix. 2. component which is causing a valid .t file to fail in regression. I have used my best judgement here to move the bug to right component. In case of ambiguity, I have placed the blame on the .t file component. Please consider test bugs under the same backlog list that tracks other bugs in your component.
REVIEW: http://review.gluster.org/13663 (georep: tests for logrotate, create+rename, hard-link rename) posted (#1) for review on master by Milind Changire (mchangir)
REVIEW: http://review.gluster.org/13663 (georep: tests for logrotate, create+rename, hard-link rename) posted (#2) for review on master by Milind Changire (mchangir)
COMMIT: http://review.gluster.org/13663 committed in master by Aravinda VK (avishwan) ------ commit 76401324af5a5e7dc3acf8d4fbe4884d6c3f0281 Author: Milind Changire <mchangir> Date: Thu Mar 10 13:45:52 2016 +0530 georep: tests for logrotate, create+rename, hard-link rename * log rotate eg. with rotate count as 2 and with the following files created x.log, x.log.1 and x.log.2, another iteration of logrotate results in the following operations to be performed x.log.2 is renamed to x.log.3 x.log.1 is renamed to x.log.2 x.log is renamed to x.log.1 x.log.3 is deleted x.log is created [possible gfid allocated that belonged to x.log.3] when a file is created, there's a big possibility that a gfid used earlier can be reassigned and reused. This causes RENAME operations to fail on slave nodes when logs are replayed, typically on a Georep restart. A function called logrotate_simulate has been added to tests/geo-rep.rc to help simulate this situation. Starting from a clean state, logrotate_simulate has to be called 4 times with a rotate count of 2 to help the logs roll over and cause a delete at the end and a gfid reallocation. With the bug-fix in place, this test should not cause the Georep session to go into a Faulty state. * create+rename On log replay after Georep restart, a create+rename causes an entry to be created for the original file, but it cannot be linked to the gfid back-end since it is associated with the renamed file before the Georep restart. A function create_rename simulates the create+rename scenario and the function create_rename_ok tests if the dangling entry is present at the slave mount. With the bug-fix in place, a dangling entry with the original name should not be found at the slave mount after logs are replayed after a Georep restart. * hard-link rename This case is similar to the create+rename case except that this is a case of renaming hard-link to one of its other names. A function hardlink_rename has been added to tests/geo-rep.rc which simulates the creation and rename of hard-link. After a Georep session restart, the test function hardlink_rename_ok should not find the source link name on the slave. With the bug-fix in place, this test should not fail. If changelogs have been enabled on the slave as well, then the logs should show an UNLINK entry for the source link name, since a rename operation of a hard link to one of its names essentially just drops the source name as per the 'mv' command semantics. Change-Id: I85b196c00cf79a11bada25ef2fe5f1dc5c0c858a BUG: 1316389 Signed-off-by: Milind Changire <mchangir> Reviewed-on: http://review.gluster.org/13663 Smoke: Gluster Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.org> NetBSD-regression: NetBSD Build System <jenkins.org> Reviewed-by: Aravinda VK <avishwan>
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.9.0, please open a new bug report. glusterfs-3.9.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://lists.gluster.org/pipermail/gluster-users/2016-November/029281.html [2] https://www.gluster.org/pipermail/gluster-users/