Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be available on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.
Bug 1277316 - Data Tiering: fix lookup-unhashed for tiered volumes.
Summary: Data Tiering: fix lookup-unhashed for tiered volumes.
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: tier
Version: rhgs-3.1
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: RHGS 3.1.2
Assignee: Dan Lambright
QA Contact: Nag Pavan Chilakam
Depends On:
Blocks: 1260783 1277716 1278603
TreeView+ depends on / blocked
Reported: 2015-11-03 01:46 UTC by Dan Lambright
Modified: 2016-09-17 15:42 UTC (History)
6 users (show)

Fixed In Version: glusterfs-3.7.5-6
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1277716 (view as bug list)
Last Closed: 2016-03-01 05:50:21 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:0193 0 normal SHIPPED_LIVE Red Hat Gluster Storage 3.1 update 2 2016-03-01 10:20:36 UTC

Description Dan Lambright 2015-11-03 01:46:01 UTC
Description of problem:

The lookup-unhashed option did not work on tiered volumes.
Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. set lookup-unhashed to on for a tiered volume.
2. observe count of lookups to server.

Actual results:

count of lookups to server is unchanged.

Expected results:

count of lookups to server should decrease.

Additional info:

Comment 3 Nag Pavan Chilakam 2015-12-11 09:07:17 UTC
Dan, Can you please provide more information on what QE needs to do as part of testing?

Comment 4 Dan Lambright 2015-12-22 16:45:53 UTC
Discussed in scrum, documenting here. 

The fix removes unnecessary work from detach tier related to the lookup-unhashed optimization. It can be tested by verifying lookup-optimization works before and after the detach tier operation. The DHT team can provide a procedure for validating the lookup-optimization feature.

Comment 5 Dan Lambright 2015-12-31 00:08:13 UTC
How to test:

1. become familiar with how lookup-unhashed works.
2. verify it works with the patch loaded , before and after detaching a tier.


lookup-unhashed improves the performance of create operations. When you create a file, DHT normally must look on all sub volumes to see if the file already exists. This is called "lookup everywhere". In the logs, you will see "dht_lookup_everywhere". With the setting enabled, you should not see this message. Instead, you would only see dht_lookup to the hashed sub volume. 

Note: try it on a subdirectory. Do not try it on the root directory (for reasons related to DHT and beyond the scope of this fix).

Do not forget to set diagnostics.client-log-level to DEBUG

on a distributed replicated volume, with cluster.lookup-optimize set to off,

$ mkdir /mnt/test1
$ touch  /mnt/test1/file

You should see "dht_lookup_everywhere" 

Now set cluster.lookup-optimize set to on. Make a *new* directory and create a file in it. It has to be new, for internal DHT reasons beyond the scope of this fix.

$ mkdir /mnt/test2
$ touch /mnt/test2/file

This time, you should *not* see dht_lookup_everywhere.


This bug's patch optimized the detach process for tiered volumes to avoid unnecessary steps. We need to confirm the patch did not break the lookup unhashed optimization. So repeat the second part of the above test, before and after detaching a tier.

On a tiered volume, with cluster.lookup-optimize set to on,

$ mkdir /mnt/test3
$ touch /mnt/test3/file

Confirm you do *not* see dht_lookup_everywhere in the logs.

Detach the tier.

Create another file in the same directory. 

$ touch /mnt/test3/file2

Confirm you do *not* see dht_lookup_everywhere in the logs.

Comment 6 Nag Pavan Chilakam 2016-01-20 11:41:13 UTC
have run tiering basic set of attach tier regression and rebalance related which passed.
Hence moving to pass

Comment 8 errata-xmlrpc 2016-03-01 05:50:21 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.