Bug 1046603
| Summary: | BVT: Rebalance failed for files with xattrs | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Lalatendu Mohanty <lmohanty> | ||||||||
| Component: | glusterd | Assignee: | Bug Updates Notification Mailing List <rhs-bugs> | ||||||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Lalatendu Mohanty <lmohanty> | ||||||||
| Severity: | low | Docs Contact: | |||||||||
| Priority: | unspecified | ||||||||||
| Version: | 2.1 | CC: | lmohanty, pkarampu, shmohan, vagarwal, vbellur | ||||||||
| Target Milestone: | --- | ||||||||||
| Target Release: | --- | ||||||||||
| Hardware: | Unspecified | ||||||||||
| OS: | Unspecified | ||||||||||
| Whiteboard: | |||||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||||
| Doc Text: | Story Points: | --- | |||||||||
| Clone Of: | Environment: | ||||||||||
| Last Closed: | 2014-02-07 06:45:45 UTC | Type: | Bug | ||||||||
| Regression: | --- | Mount Type: | --- | ||||||||
| Documentation: | --- | CRM: | |||||||||
| Verified Versions: | Category: | --- | |||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||
| Embargoed: | |||||||||||
| Attachments: |
|
||||||||||
Created attachment 841817 [details]
Rebalance logs from masternode
Can we manually recreate this issue? Today's BVT run does not have this issue. The same test case passed in the test run. The yesterday's BVT run and today's BVT run used the same build for the tests as there were no code merged in between the two tests. Today's BVT run: https://beaker.engineering.redhat.com/jobs/566272 Yesterday's BVT Run: https://beaker.engineering.redhat.com/jobs/566031 It seems the issue is an intermittent one as it is not always reproducible. Lala,
According to the logs the exit status of rebalance is "[2013-12-25 07:19:43.822761] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 100, failures: 0, skipped: 17
". Which contradicts whatever is shown. I just want to make sure that this is indeed the log file of the run before starting debugging.
Pranith
Pranith,
I have downloaded rebalance logs for the same volume from all three nodes used in this test and checked if there are failures in the log but didn't find any.
$ ls
hosdu-rebalance.log.masternode hosdu-rebalance.log.node hosdu-rebalance.log.peer
$ grep -r "failures"
hosdu-rebalance.log.node:[2013-12-25 07:13:21.348018] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 64, failures: 0, skipped: 0
hosdu-rebalance.log.node:[2013-12-25 07:13:21.684641] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 101, failures: 0, skipped: 0
hosdu-rebalance.log.node:[2013-12-25 07:19:43.381880] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 77, failures: 0, skipped: 0
hosdu-rebalance.log.node:[2013-12-25 07:19:43.564128] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 100, failures: 0, skipped: 0
hosdu-rebalance.log.peer:[2013-12-25 00:34:27.548474] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 63, failures: 0, skipped: 0
hosdu-rebalance.log.peer:[2013-12-25 00:34:27.883724] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 101, failures: 0, skipped: 0
hosdu-rebalance.log.peer:[2013-12-25 00:40:49.599169] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 46, failures: 0, skipped: 0
hosdu-rebalance.log.peer:[2013-12-25 00:40:53.869961] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 15, size: 196608000, lookups: 115, failures: 0, skipped: 0
hosdu-rebalance.log.masternode:[2013-12-25 07:13:21.338627] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 3, failures: 0, skipped: 0
hosdu-rebalance.log.masternode:[2013-12-25 07:13:24.682417] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 7, size: 91750400, lookups: 107, failures: 0, skipped: 25
hosdu-rebalance.log.masternode:[2013-12-25 07:19:43.371939] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 47, failures: 0, skipped: 10
hosdu-rebalance.log.masternode:[2013-12-25 07:19:43.822761] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 100, failures: 0, skipped: 17
I am attaching the logs for your reference.
About creating the extended attributes, below the function which creates files and adds extended attributes during the test.
create_xattrs_168896()
{
echo "${FUNCNAME} - Creates files with xattrs set"
for i in `seq 1 50`
do
mkdir -p $MOUNT_POINT/xattr
touch $MOUNT_POINT/xattr/xattr-color${i}.file
setfattr -n user.color -v "green" $MOUNT_POINT/xattr/xattr-color${i}.file
touch $MOUNT_POINT/xattr/xattr-own${i}.file
setfattr -n user.do_i_own -v "YES" $MOUNT_POINT/xattr/xattr-own${i}.file
touch $MOUNT_POINT/xattr/xattr-user${i}.file
setfattr -n user.myname -v "ben" $MOUNT_POINT/xattr/xattr-user${i}.file
done
}
Created attachment 841881 [details]
Rebalance logs from peer
Created attachment 841882 [details]
Rebalance logs from node
Lala, According to Comment-6, number of files migrated is 15+7, where as according to output in the bug it is 19+11. So it is still puzzling :-(. Pranith Pranith, I am looking in to it. I will update if I find more information. Not able to reproduce the issue manually. Also BVT is not reporting the issue in recent runs. Reducing the severity of the issue to low. Will close the issue if I dont see the issue for some more days. This issue is not seen with last BVT runs(in last couple of weeks). Hence closing the bug. |
Description of problem: Rebalance failed for files with extended attributes. 02:50:57 ] :: The rebalance is complete, logging final status Node Rebalanced-files size scanned failures skipped status run time in secs --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 19 0Bytes 174 0 0 completed 1.00 rhsauto008.lab.eng.blr.redhat.com 21 0Bytes 189 1 0 completed 1.00 rhsauto022.lab.eng.blr.redhat.com 0 0Bytes 170 0 0 completed 1.00 volume rebalance: rebalvol: success: :: [ FAIL ] :: There were errors with the rebalance, please see logs. \ Version-Release number of selected component (if applicable): glusterfs-server-3.4.0.52rhs-1387920125.el6.x86_64.rpm How reproducible: Found in BVT run. Haven't tried manually. If required I will try to reproduce it manually Steps to Reproduce: 1. Create Gluster volume (distribute-replicate). 2. Mount it on a Client using native fuse mount. 3. Create files with extended attributes. 4. Add brick and start rebalance Actual results: Rebalance status have failure Expected results: There shouldn't be any failure. There may be files in skipped but not in failure Additional info: