Bug 764664 (GLUSTER-2932)

Summary: Writing "pure path resolution ..." log message when creating linkfile influence on performance
Product: [Community] GlusterFS Reporter: hz02ruc
Component: distributeAssignee: shishir gowda <sgowda>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: medium Docs Contact:
Priority: medium    
Version: 3.1.3CC: amarts, gluster-bugs, lakshmipathi, nsathyan
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: RTNR Mount Type: fuse
Documentation: --- CRM:
Verified Versions: 3.2.3 Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Amar Tumballi 2011-05-24 05:04:21 UTC
patch http://patches.gluster.com/patch/7108/ fixes this. will be available in 3.2.1 release (its already committed to git master branch).

Comment 1 hz02ruc 2011-05-24 07:50:15 UTC
BACKGROUND:
        We are using GlusterFS 3.1.3 as our storage system. Recently
        A brick was removed from our system which stores about 50TB data, 
        then the IO performance of the storage nodes declined very much.
    
        I noticed that the log files of storage nodes grows fast, i have to  
        truncate them until they reached above 1GB. The log files are full of
        warning messages "[server-resolve.c:server_resolve]pure path resolution for <path>".

        It seems that only linkfiles can cause this log message, and i did a
        test to reproduce the problem.

Reproduce:
        4 server process, each export a directory
                server : ~/share
                server2: ~/share2
                server3: ~/share3
                server4: ~/share4

        client: configured with dht 
                mountpoint: ~/mnt

    
        1. create 4 files.
    
        huz@furutuki:~/dht$ touch ~/mnt/tags-1 ~/mnt/tags-2 ~/mnt/tags-6 ~/mnt/tags-11
        huz@furutuki:~/dht$ ll ~/share
        total 0
        -rw-r--r-- 1 huz huz 0 2011-05-24 14:37 tags-1
        huz@furutuki:~/dht$ ll ~/share2
        total 0
        -rw-r--r-- 1 huz huz 0 2011-05-24 14:37 tags-2
        huz@furutuki:~/dht$ ll ~/share3
        total 0
        -rw-r--r-- 1 huz huz 0 2011-05-24 14:37 tags-6
        huz@furutuki:~/dht$ ll ~/share4
        total 0
        -rw-r--r-- 1 huz huz 0 2011-05-24 14:37 tags-11
        huz@furutuki:~/dht$ ll ~/mnt
        total 0

        -rw-r--r-- 1 huz huz 0 2011-05-24 14:37 tags-1
        -rw-r--r-- 1 huz huz 0 2011-05-24 14:37 tags-11
        -rw-r--r-- 1 huz huz 0 2011-05-24 14:37 tags-2
        -rw-r--r-- 1 huz huz 0 2011-05-24 14:37 tags-6

        2. remove brick4 and reconfigure

        huz@furutuki:~/dht$ ps -ef | grep glusterfs
        root     13762     1  0 14:36 ?        00:00:00 glusterfsd -f /home/huz/dht/server.vol -l /home/huz/dht/server.log 
        root     13767     1  0 14:36 ?        00:00:00 glusterfsd -f /home/huz/dht/server2.vol -l /home/huz/dht/server2.log 
        root     13772     1  0 14:36 ?        00:00:00 glusterfsd -f /home/huz/dht/server3.vol -l /home/huz/dht/server3.log 
        root     13777     1  0 14:36 ?        00:00:00 glusterfsd -f /home/huz/dht/server4.vol -l /home/huz/dht/server4.log
        root     13787     1  0 14:36 ?        00:00:00 glusterfs -f /home/huz/dht/client.vol -l /home/huz/dht/client.log/home/huz/mnt
        huz@furutuki:~/dht$ vi client.vol
        huz@furutuki:~/dht$ sudo kill -s TERM 13777
        huz@furutuki:~/dht$ sudo kill -s HUP 13787

        3. ls ~/mnt (lookup) and check that linkfiles are created(brick2, brick3)
        huz@furutuki:~/dht$ ll ~/mnt
        total 0
        -rw-r--r-- 1 huz huz 0 2011-05-24 14:37 tags-1
        -rw-r--r-- 1 huz huz 0 2011-05-24 14:37 tags-2
        -rw-r--r-- 1 huz huz 0 2011-05-24 14:37 tags-6
        huz@furutuki:~/dht$ ll ~/share
        total 0
        -rw-r--r-- 1 huz huz 0 2011-05-24 14:37 tags-1
        huz@furutuki:~/dht$ ll ~/share2
        total 0
        ---------T 1 huz huz 0 2011-05-24 14:38 tags-1
        -rw-r--r-- 1 huz huz 0 2011-05-24 14:37 tags-2
        huz@furutuki:~/dht$ ll ~/share3
        total 0
        ---------T 1 huz huz 0 2011-05-24 14:38 tags-2
        -rw-r--r-- 1 huz huz 0 2011-05-24 14:37 tags-6

        4. "pure path resolution for <path> (SETXATTR)" are wrote in the log file: server2.log server3.log

-------------------------------------

       It seems in function server_resolve(), gfid & paragfid == null can lead such a message.

        http://patches.gluster.com/patch/4504/ set gfid in *fuse layer* for new entries, but
        in my test, after removing brick 4 and reconfigure, MKNOD will be called in *dht layer*
        in dht_linkfile_create() to create linkfile and call SETXATTR in its callback function,
        but no gfid is set before, so in server path resolution function, the log message outputs.

        By the way,I searched the web, this log message was added in 3.1.X and the loglevel
        was changed  to "INFO" in GlusterFS 3.2.0. I really want to know why the dev team maded
        these changes.

        In order to avoid the problem, i need to change the loglevel  to "TRACE", "DEBUG" or
        delete this log message for the present.

        it's appreciated if you can give me any advise and tell me whether my understanding
        is correct or not.

        Thanks in advance.

Comment 2 Lakshmipathi G 2011-08-30 04:32:34 UTC
tested with 3.2.3.  tried to create few files with 'T' - log file (with default log level)doesn't show message like pure path resoultion.