Bug 761926 (GLUSTER-194)

Pasting from gluster-users(Shehjar):


The behavior is still the same(Ed: meaning with 2.0.6rc2(Shehjar))

[root@compute-0-0 ~]# wget -O /dev/null --header "Host: www.myhost.local" http://c0-3/cafe/siam/topic/F7800428/F7800428.html
--12:20:33--  http://c0-3/cafe/siam/topic/F7800428/F7800428.html
Resolving c0-3... 10.1.255.251
Connecting to c0-3|10.1.255.251|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 264373 (258K) [text/html]
Saving to: `/dev/null'

100%[====================================================================================================================>] 264,373     --.-K/s   in 0.003s

12:20:33 (81.8 MB/s) - `/dev/null' saved [264373/264373]

[root@compute-0-0 ~]# wget -O /dev/null --header "Host: www.myhost.local" http://c0-3/cafe/siam/topic/F7800428/F7800428.html
--12:20:35--  http://c0-3/cafe/siam/topic/F7800428/F7800428.html
Resolving c0-3... 10.1.255.251
Connecting to c0-3|10.1.255.251|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 0 [text/html]
Saving to: `/dev/null'

    [ <=>                                                                                                                 ] 0           --.-K/s   in 0s    

12:20:35 (0.00 B/s) - `/dev/null' saved [0/0]

[root@compute-0-0 ~]# wget -O /dev/null --header "Host: www.myhost.local" http://c0-3/cafe/siam/topic/F7800428/F7800428.html
--12:20:38--  http://c0-3/cafe/siam/topic/F7800428/F7800428.html
Resolving c0-3... 10.1.255.251
Connecting to c0-3|10.1.255.251|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 0 [text/html]
Saving to: `/dev/null'

    [ <=>                                                                                                                 ] 0           --.-K/s   in 0s    

12:20:38 (0.00 B/s) - `/dev/null' saved [0/0]

[root@compute-0-0 ~]# wget -O /dev/null --header "Host: www.myhost.local" http://c0-3/cafe/siam/topic/F7800428/F7800428.html
--12:20:39--  http://c0-3/cafe/siam/topic/F7800428/F7800428.html
Resolving c0-3... 10.1.255.251
Connecting to c0-3|10.1.255.251|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 264373 (258K) [text/html]
Saving to: `/dev/null'

100%[====================================================================================================================>] 264,373     --.-K/s   in 0.003s

12:20:39 (82.3 MB/s) - `/dev/null' saved [264373/264373]

[root@compute-0-0 ~]#


4 consequtive wget yield different content length. Note that, this only happened with worker + booster, prefork has no problem.

And the web server seems to use correct version of glusterfs

[root@compute-0-3 ~]# glusterfs -V
glusterfs 2.0.6rc2 built on Aug  6 2009 12:02:03
Repository revision: v2.0.5-13-g5e3ca25
Copyright (c) 2006-2009 Z RESEARCH Inc. <http://www.zresearch.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General Public License.
[root@compute-0-3 ~]#

And all glusterfsd too

[root@fe1 ~]# pexec -p storage glusterfs -V \| grep 2\.0\.6
compute-0-6:    glusterfs 2.0.6rc2 built on Aug  6 2009 12:02:03
compute-0-8:    glusterfs 2.0.6rc2 built on Aug  6 2009 12:02:03
compute-0-9:    glusterfs 2.0.6rc2 built on Aug  6 2009 12:02:03
compute-0-7:    glusterfs 2.0.6rc2 built on Aug  6 2009 12:02:03
[root@fe1 ~]#

Don't sure if there is anyway to query version from running glusterfsd, but I did /etc/init.d/glusterfsd restart already.

2009/8/3 Shehjar Tikoo <shehjart>

    Somsak Sriprayoonsakul wrote:

        Ok, I have a chance to run booster over 2.0.4



    Have you tried configuring booster with the help doc available at:
    http://www.gluster.org/docs/index.php/BoosterConfiguration

    -Shehjar


        Please find the attach file for my configuration

        I did configure boost and try simple ls over my Gluster file system. Here's
        the output of ls -al

        [root@compute-0-3 ~]# LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so
        ls -l /gluster/www/
        ls: /gluster/www/: Invalid argument
        ls: /gluster/www/members.pantip.com: Invalid argument
        ls: /gluster/www/cafe.pantip.com: Invalid argument
        ls: /gluster/www/admin.pantip.com: Invalid argument
        ls: /gluster/www/www.pantip.com: Invalid argument
        ls: /gluster/www/passwd3.sql: Invalid argument
        ls: /gluster/www/passwd.sql: Invalid argument
        ls: closing directory /gluster/www/: File descriptor in bad state
        total 129972
        drwxr-xr-x  3 root   root       8192 May 11 16:13 admin.pantip.com
        drwxr-xr-x  5 root   root       8192 May 18 11:11 cafe.pantip.com
        drwxr-xr-x  3 root   root       8192 May 11 18:48 members.pantip.com
        -rw-r--r--  1 root   root   66654820 May 18 10:50 passwd3.sql
        -rw-r--r--  1 root   root   66225769 May 18 10:33 passwd.sql
        drwxr-xr-x 11 apache apache     8192 May 18 09:47 www.pantip.com
        [root@compute-0-3 ~]#

        [root@compute-0-3 ~]# LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so
        cp /etc/issue /gluster/
        [root@compute-0-3 ~]# LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so
        ls -l /gluster/issue
        ls: /gluster/issue: Invalid argument
        -rw-r--r-- 1 root root 47 Aug  2 14:57 /gluster/issue
        [root@compute-0-3 ~]# LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so
        cat /gluster/issue
        CentOS release 5.3 (Final)
        Kernel \r on an \m

        [root@compute-0-3 ~]#


        Despite all those errors, output seems to be fine

        And this is what inside my booster.log

        [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
        libglusterfsclient: invalid argument: value
        [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
        libglusterfsclient: invalid argument: value
        [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
        libglusterfsclient: invalid argument: value
        [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
        libglusterfsclient: invalid argument: value
        [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
        libglusterfsclient: invalid argument: value
        [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
        libglusterfsclient: invalid argument: value
        [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
        libglusterfsclient: invalid argument: value
        [2009-08-02 14:56:27] E [libglusterfsclient.c:4194:__glusterfs_stat]
        libglusterfsclient: path lookup failed for (/hosts)
        [2009-08-02 14:56:37] E [libglusterfsclient.c:2244:glusterfs_getxattr]
        libglusterfsclient: invalid argument: value
        [2009-08-02 14:57:00] E [libglusterfsclient.c:4194:__glusterfs_stat]
        libglusterfsclient: path lookup failed for (/issue)
        [2009-08-02 14:57:07] E [libglusterfsclient.c:2244:glusterfs_getxattr]
        libglusterfsclient: invalid argument: value

        Then, I try to LD_PRELOAD apache (prefork). I change the target from
        /gluster to /usr/home instead (the web application needs it). Then I tried
        to strace the httpd process and found that httpd crash at the points where
        httpd tried to read configuration file stored on Gluster volume (bad file
        descriptor). I will try to move this configuration file some other places
        and test again.

        2009/7/31 Raghavendra G <raghavendra>

            Hi,

            On Thu, Jul 30, 2009 at 11:39 AM, Somsak
            Sriprayoonsakul<somsaks> wrote:

                Thank you very much for you reply

                At the time we used 2.0.3, and yes we used stock Apache from CentOS. I

            will

                try 2.0.4 very soon to see if it's work.

                For Booster, it seems not working correctly for me. Booster complains a

            lots

                of error with plain 'ls' command (but giving the correct output). Also,

            with

            Can you mail those errors?

                booster, Apache process refuse to start. I will try 2.0.4 to see if it
                improves. If not, I will attach error log next time.

            logs are very much appreciated.


                2009/7/30 Raghavendra G <raghavendra>

                    Hi Somsak,

                    Sorry for the delayed reply. Below you've mentioned that you've problems
                    with apache and booster. Going forward, Apache over booster will be the
                    preferred approach. Can you tell us what version of glusterfs you are

            using?

                    And as I can understand you are using apache 2.2, am I correct?

                    regards,
                    ----- Original Message -----
                    From: "Liam Slusser" <lslusser>
                    To: "Somsak Sriprayoonsakul" <somsaks>
                    Cc: gluster-users
                    Sent: Saturday, July 25, 2009 3:46:14 AM GMT +04:00 Abu Dhabi / Muscat
                    Subject: Re: [Gluster-users] Gluster 2.0.3 + Apache on CentOS5

            performance

                     issue

                    I haven't tried an apples to apples comparison with Apache+mod_gluster

            vs

                    Apache+fuse+gluster however i do run both setups.  I load tested both
                    setups
                    so to verified it could handle 4x our normal daily load and left it at
                    that.
                     I didn't actually compare the two (although that might be cool to do
                    someday).
                    I really like the idea of Apache+mod_gluster as I don't have to deal

            with

                    the whole fuse and mounting the filesystem.  It always scares me having

            a

                    public facing webserver with your whole backend fileshare mounted

            locally.

                     Its very slick for serving content such as media files.  We serve audio
                    content to our CDN with a pair of Apache/mod_gluster servers - pushing
                    200-300mbit on average daily and everything works very well.

                    We run an apache+fuse+gluster setup because we need to run some mod_perl
                    before serving the actual content.  However performance is still very
                    good.
                     We do around 50-100 requests (all jpeg images) per second off of a fuse
                    mount and everything works great.  We also have a java

            tomcat+fuse+gluster

                    service which does image manipulation on the fly off of a gluster mount.

                    We have two backend gluster servers using replication which serve all

            this

                    content.

                    If you would like more information on our setup id be happy to share
                    offline.  Just email me privately.

                    thanks,
                    liam

                    On Fri, Jul 24, 2009 at 8:08 AM, Somsak Sriprayoonsakul
                    <somsaks>wrote:

                        Oh thank you, thought noone will reply me :)

                        Have you tried Apache + Fuse over GlusterFS? How is the performance?

                        Also, anyone in this mailing-list have tried Apache with booster? I
                        tried
                        it but Apache refuse to start (just hang and freeze).

                        2009/7/23 Liam Slusser <lslusser>


                            We use mod_gluster and Apache
                            2.2 with good results.  We also ran into the same issue as you that

            we

                            ran out of memory past 150 threads even on a 8gig machine.  We got

            around

                            this by compiling Apache using mpm-worker
                            (threads) instead of prefork - it uses 1/4 as much ram with the same
                            number
                            of connections (150-200) and everything has been running smoothly.  I
                            cannot
                            see any performance difference except it using way less memory.
                            liam


                            On Sun, Jul 12, 2009 at 5:11 AM, Somsak Sriprayoonsakul <
                            somsaks> wrote:

                                Hello,

                                We have been evaluating the choice for the new platform for a

            webboard

                                system.
                                The webboard is PHP scripts that generate/modify HTML page when user
                                posting/add comment to the page, resulting topic is actually stored

            as

                                a
                                HTML file with all related file (file attach to the topic, etc..
                                )stored in
                                its own directory for each topic. In general, the web site mostly
                                serve a
                                lot of small static files using Apache while using PHP to do other
                                dynamic
                                contents. This system has been working very well in the past, with

            the

                                increasing page view rate, it is very likely that we will need some
                                kind of
                                Cluster file system as backend very soon.

                                We have set up a test system using Grinder as stress test tool. The
                                test
                                system is 11 machines of Intel Dual Core x86_64 CentOS5 with stock
                                Apache
                                (prefork, since the goal is to use this with PHP), linked together
                                with
                                Gigabit Ethernet. We try to compare the performance of either using
                                single
                                NFS server in sync mode against using 4 Gluster nodes (distribute of

            2

                                replicated nodes) through Fuse. However, the transaction per second
                                (TPS)
                                result is not good.

                                NFS (single server, sync mode)
                                 - 100 thread of client - Peak TPS = 1716.67, Avg. TPS = 1066, mean
                                response time = 61.63 ms
                                 - 200 threads - Peak TPS = 2790, Avg. TPS = 1716, mean rt = 87.33

            ms

                                 - 400 threads - Peak TPS = 3810, Avg. TPS = 1800, mean rt = 165ms
                                 - 600 threads - Peak TPS = 4506.67, Avg. TPS = 1676.67, mean rt =
                                287.33ms

                                4 nodes Gluster (2 distribute of replicated 2 node)
                                - 100 thread - peak TPS = 1293.33, Avg. TPS = 430, mean rt =

            207.33ms

                                - 200 threads - Peak TPS = 974.67, Avg. TPS = 245.33, mean rt =
                                672.67ms
                                - 300 threads - Peak TPS = 861.33, Avg. TPS = 210, mean rt = 931.33
                                (no 400-600 threads since we run out of client machine, sorry).

                                gfsd is configured to use 32 thread of iothread as brick. gfs-client
                                is
                                configured to use
                                io-cache->write-behind->readahead->distribute->replicate.
                                io-cache cache-size is 256MB. I used patched Fuse downloaded from
                                Gluster
                                web-site (build through DKMS).

                                As the result yield, it seems that Gluster performance worse with
                                increasing no. of client. One observation is that the glusterfs
                                process on
                                client is taking about 100% of CPU during all the tests. glusterfsd

            is

                                utilizing only 70-80% of CPUs during the test time. Note that system
                                is Dual
                                core.

                                I also tried using modglusterfs and not using fuse at all to serve

            all

                                the static files and conduct another test with Grinder. The result

            is

                                about
                                the same, 1000+ peak TPS with 2-400 avg. TPS. A problem arise in

            this

                                test
                                that each Apache prefork process used more about twice more memory

            and

                                we
                                need to lower number of httpd processes by about half.

                                I tried disable EnableMMAP and it didn't help much. Adjusting
                                readahead,
                                write behind according to GlusterOptimization page didn't help much
                                either.

                                My question is, there seems to be bottleneck in this setup, but how
                                can I
                                track this? Note that, I didn't do any other optimization other than
                                what
                                said above. Are there any best practice configuration for using

            Apache

                                to
                                serve a bunch of small static files like this around?

                                Regards,

                                Somsak




                                _______________________________________________
                                Gluster-users mailing list
                                Gluster-users
                                http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users


                    _______________________________________________
                    Gluster-users mailing list
                    Gluster-users
                    http://gluster.org/cgi-bin/mailman/listinfo/gluster-users



            regards,
            --
            Raghavendra G



        ------------------------------------------------------------------------


        _______________________________________________
        Gluster-users mailing list
        Gluster-users
        http://gluster.org/cgi-bin/mailman/listinfo/gluster-users