Tests that are retried get their logs tarball overwritten by the new instance, thus making it difficult to debug the initial failure. The framework should retain the older tarball, to aid debugging and addressing failures.
REVIEW: https://review.gluster.org/20682 (tests: Add ability to preserve older tarball for retried tests) posted (#1) for review on master by Shyamsundar Ranganathan
COMMIT: https://review.gluster.org/20682 committed in master by "Atin Mukherjee" <amukherj> with a commit message- tests: Add ability to preserve older tarball for retried tests When a test is retried, the cleanup directives overwrite the older tarball with the latest one, thus losing the logs from the failed run. This patch changes run-tests.sh to rename the older tarball when retrying a test, thus preserving the same. The tarball is renamed using a time stamp and optionally a trailing sequence number, in case the test fails within the very second. Although the sequence # is not strictly required as we retry only once, it provides a defence for any future enhancements to the same. Fixes: bz#1614062 Change-Id: I9afe486b0b6f6a26f2ad0642e38bc0ba15b3ecc9 Signed-off-by: ShyamsundarR <srangana>
Moving this back to assigned, as we still need more fixes to move the tarball generation outside of include.rc:cleanup to run-tests.sh. See conversation in the patch posted at comment #2
REVIEW: https://review.gluster.org/20740 (tests: Preserve tarball of tests when they timeout) posted (#1) for review on master by Shyamsundar Ranganathan
COMMIT: https://review.gluster.org/20740 committed in master by "Atin Mukherjee" <amukherj> with a commit message- tests: Preserve tarball of tests when they timeout When tests timeout, the timeout command sends TERM signal to the command being executed. In the case of run-tests.sh it invokes prove, which further invokes perl and finally the test is run using bash. The TERM signal does not seem to be reachnig the end bash that is actually executing the tests, and hence when any test is terminated due to a timeout, the cleanup routine in include.rc does not get a chance to run and preserve the tarball. Further, cleanup invokes tarball generation, but is invoked at the beginning and end of every test, and at times in beteween as well. This caused way too many tarballs in case we decide to preserve the same whenever generated by cleanup. This patch hence moves the tarball generation to run-tests.sh instead, and further stores them named <test>-iteration-<n>.tar and also prints tarball name generated and stored per iteration. This should help relate failed runs to the tarball iteration # and to look at relevant logs. Further the patch also provides a -p option to run-tests.sh for unit testing purposes, where running a test in a loop without the option will generate as many tarballs, and using the option will reduce this to preserving the last tarball, saving space in smaller unit test setups. Fixes: bz#1614062 Change-Id: I0aee76c89df0691cf4d0c1fcd4c04dffe0d7c896 Signed-off-by: ShyamsundarR <srangana>
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-5.0, please open a new bug report. glusterfs-5.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] https://lists.gluster.org/pipermail/announce/2018-October/000115.html [2] https://www.gluster.org/pipermail/gluster-users/