Hide Forgot
Pasting from gluster-users(Shehjar): The behavior is still the same(Ed: meaning with 2.0.6rc2(Shehjar)) [root@compute-0-0 ~]# wget -O /dev/null --header "Host: www.myhost.local" http://c0-3/cafe/siam/topic/F7800428/F7800428.html --12:20:33-- http://c0-3/cafe/siam/topic/F7800428/F7800428.html Resolving c0-3... 10.1.255.251 Connecting to c0-3|10.1.255.251|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 264373 (258K) [text/html] Saving to: `/dev/null' 100%[====================================================================================================================>] 264,373 --.-K/s in 0.003s 12:20:33 (81.8 MB/s) - `/dev/null' saved [264373/264373] [root@compute-0-0 ~]# wget -O /dev/null --header "Host: www.myhost.local" http://c0-3/cafe/siam/topic/F7800428/F7800428.html --12:20:35-- http://c0-3/cafe/siam/topic/F7800428/F7800428.html Resolving c0-3... 10.1.255.251 Connecting to c0-3|10.1.255.251|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 0 [text/html] Saving to: `/dev/null' [ <=> ] 0 --.-K/s in 0s 12:20:35 (0.00 B/s) - `/dev/null' saved [0/0] [root@compute-0-0 ~]# wget -O /dev/null --header "Host: www.myhost.local" http://c0-3/cafe/siam/topic/F7800428/F7800428.html --12:20:38-- http://c0-3/cafe/siam/topic/F7800428/F7800428.html Resolving c0-3... 10.1.255.251 Connecting to c0-3|10.1.255.251|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 0 [text/html] Saving to: `/dev/null' [ <=> ] 0 --.-K/s in 0s 12:20:38 (0.00 B/s) - `/dev/null' saved [0/0] [root@compute-0-0 ~]# wget -O /dev/null --header "Host: www.myhost.local" http://c0-3/cafe/siam/topic/F7800428/F7800428.html --12:20:39-- http://c0-3/cafe/siam/topic/F7800428/F7800428.html Resolving c0-3... 10.1.255.251 Connecting to c0-3|10.1.255.251|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 264373 (258K) [text/html] Saving to: `/dev/null' 100%[====================================================================================================================>] 264,373 --.-K/s in 0.003s 12:20:39 (82.3 MB/s) - `/dev/null' saved [264373/264373] [root@compute-0-0 ~]# 4 consequtive wget yield different content length. Note that, this only happened with worker + booster, prefork has no problem. And the web server seems to use correct version of glusterfs [root@compute-0-3 ~]# glusterfs -V glusterfs 2.0.6rc2 built on Aug 6 2009 12:02:03 Repository revision: v2.0.5-13-g5e3ca25 Copyright (c) 2006-2009 Z RESEARCH Inc. <http://www.zresearch.com> GlusterFS comes with ABSOLUTELY NO WARRANTY. You may redistribute copies of GlusterFS under the terms of the GNU General Public License. [root@compute-0-3 ~]# And all glusterfsd too [root@fe1 ~]# pexec -p storage glusterfs -V \| grep 2\.0\.6 compute-0-6: glusterfs 2.0.6rc2 built on Aug 6 2009 12:02:03 compute-0-8: glusterfs 2.0.6rc2 built on Aug 6 2009 12:02:03 compute-0-9: glusterfs 2.0.6rc2 built on Aug 6 2009 12:02:03 compute-0-7: glusterfs 2.0.6rc2 built on Aug 6 2009 12:02:03 [root@fe1 ~]# Don't sure if there is anyway to query version from running glusterfsd, but I did /etc/init.d/glusterfsd restart already. 2009/8/3 Shehjar Tikoo <shehjart> Somsak Sriprayoonsakul wrote: Ok, I have a chance to run booster over 2.0.4 Have you tried configuring booster with the help doc available at: http://www.gluster.org/docs/index.php/BoosterConfiguration -Shehjar Please find the attach file for my configuration I did configure boost and try simple ls over my Gluster file system. Here's the output of ls -al [root@compute-0-3 ~]# LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so ls -l /gluster/www/ ls: /gluster/www/: Invalid argument ls: /gluster/www/members.pantip.com: Invalid argument ls: /gluster/www/cafe.pantip.com: Invalid argument ls: /gluster/www/admin.pantip.com: Invalid argument ls: /gluster/www/www.pantip.com: Invalid argument ls: /gluster/www/passwd3.sql: Invalid argument ls: /gluster/www/passwd.sql: Invalid argument ls: closing directory /gluster/www/: File descriptor in bad state total 129972 drwxr-xr-x 3 root root 8192 May 11 16:13 admin.pantip.com drwxr-xr-x 5 root root 8192 May 18 11:11 cafe.pantip.com drwxr-xr-x 3 root root 8192 May 11 18:48 members.pantip.com -rw-r--r-- 1 root root 66654820 May 18 10:50 passwd3.sql -rw-r--r-- 1 root root 66225769 May 18 10:33 passwd.sql drwxr-xr-x 11 apache apache 8192 May 18 09:47 www.pantip.com [root@compute-0-3 ~]# [root@compute-0-3 ~]# LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so cp /etc/issue /gluster/ [root@compute-0-3 ~]# LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so ls -l /gluster/issue ls: /gluster/issue: Invalid argument -rw-r--r-- 1 root root 47 Aug 2 14:57 /gluster/issue [root@compute-0-3 ~]# LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so cat /gluster/issue CentOS release 5.3 (Final) Kernel \r on an \m [root@compute-0-3 ~]# Despite all those errors, output seems to be fine And this is what inside my booster.log [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr] libglusterfsclient: invalid argument: value [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr] libglusterfsclient: invalid argument: value [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr] libglusterfsclient: invalid argument: value [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr] libglusterfsclient: invalid argument: value [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr] libglusterfsclient: invalid argument: value [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr] libglusterfsclient: invalid argument: value [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr] libglusterfsclient: invalid argument: value [2009-08-02 14:56:27] E [libglusterfsclient.c:4194:__glusterfs_stat] libglusterfsclient: path lookup failed for (/hosts) [2009-08-02 14:56:37] E [libglusterfsclient.c:2244:glusterfs_getxattr] libglusterfsclient: invalid argument: value [2009-08-02 14:57:00] E [libglusterfsclient.c:4194:__glusterfs_stat] libglusterfsclient: path lookup failed for (/issue) [2009-08-02 14:57:07] E [libglusterfsclient.c:2244:glusterfs_getxattr] libglusterfsclient: invalid argument: value Then, I try to LD_PRELOAD apache (prefork). I change the target from /gluster to /usr/home instead (the web application needs it). Then I tried to strace the httpd process and found that httpd crash at the points where httpd tried to read configuration file stored on Gluster volume (bad file descriptor). I will try to move this configuration file some other places and test again. 2009/7/31 Raghavendra G <raghavendra> Hi, On Thu, Jul 30, 2009 at 11:39 AM, Somsak Sriprayoonsakul<somsaks> wrote: Thank you very much for you reply At the time we used 2.0.3, and yes we used stock Apache from CentOS. I will try 2.0.4 very soon to see if it's work. For Booster, it seems not working correctly for me. Booster complains a lots of error with plain 'ls' command (but giving the correct output). Also, with Can you mail those errors? booster, Apache process refuse to start. I will try 2.0.4 to see if it improves. If not, I will attach error log next time. logs are very much appreciated. 2009/7/30 Raghavendra G <raghavendra> Hi Somsak, Sorry for the delayed reply. Below you've mentioned that you've problems with apache and booster. Going forward, Apache over booster will be the preferred approach. Can you tell us what version of glusterfs you are using? And as I can understand you are using apache 2.2, am I correct? regards, ----- Original Message ----- From: "Liam Slusser" <lslusser> To: "Somsak Sriprayoonsakul" <somsaks> Cc: gluster-users Sent: Saturday, July 25, 2009 3:46:14 AM GMT +04:00 Abu Dhabi / Muscat Subject: Re: [Gluster-users] Gluster 2.0.3 + Apache on CentOS5 performance issue I haven't tried an apples to apples comparison with Apache+mod_gluster vs Apache+fuse+gluster however i do run both setups. I load tested both setups so to verified it could handle 4x our normal daily load and left it at that. I didn't actually compare the two (although that might be cool to do someday). I really like the idea of Apache+mod_gluster as I don't have to deal with the whole fuse and mounting the filesystem. It always scares me having a public facing webserver with your whole backend fileshare mounted locally. Its very slick for serving content such as media files. We serve audio content to our CDN with a pair of Apache/mod_gluster servers - pushing 200-300mbit on average daily and everything works very well. We run an apache+fuse+gluster setup because we need to run some mod_perl before serving the actual content. However performance is still very good. We do around 50-100 requests (all jpeg images) per second off of a fuse mount and everything works great. We also have a java tomcat+fuse+gluster service which does image manipulation on the fly off of a gluster mount. We have two backend gluster servers using replication which serve all this content. If you would like more information on our setup id be happy to share offline. Just email me privately. thanks, liam On Fri, Jul 24, 2009 at 8:08 AM, Somsak Sriprayoonsakul <somsaks>wrote: Oh thank you, thought noone will reply me :) Have you tried Apache + Fuse over GlusterFS? How is the performance? Also, anyone in this mailing-list have tried Apache with booster? I tried it but Apache refuse to start (just hang and freeze). 2009/7/23 Liam Slusser <lslusser> We use mod_gluster and Apache 2.2 with good results. We also ran into the same issue as you that we ran out of memory past 150 threads even on a 8gig machine. We got around this by compiling Apache using mpm-worker (threads) instead of prefork - it uses 1/4 as much ram with the same number of connections (150-200) and everything has been running smoothly. I cannot see any performance difference except it using way less memory. liam On Sun, Jul 12, 2009 at 5:11 AM, Somsak Sriprayoonsakul < somsaks> wrote: Hello, We have been evaluating the choice for the new platform for a webboard system. The webboard is PHP scripts that generate/modify HTML page when user posting/add comment to the page, resulting topic is actually stored as a HTML file with all related file (file attach to the topic, etc.. )stored in its own directory for each topic. In general, the web site mostly serve a lot of small static files using Apache while using PHP to do other dynamic contents. This system has been working very well in the past, with the increasing page view rate, it is very likely that we will need some kind of Cluster file system as backend very soon. We have set up a test system using Grinder as stress test tool. The test system is 11 machines of Intel Dual Core x86_64 CentOS5 with stock Apache (prefork, since the goal is to use this with PHP), linked together with Gigabit Ethernet. We try to compare the performance of either using single NFS server in sync mode against using 4 Gluster nodes (distribute of 2 replicated nodes) through Fuse. However, the transaction per second (TPS) result is not good. NFS (single server, sync mode) - 100 thread of client - Peak TPS = 1716.67, Avg. TPS = 1066, mean response time = 61.63 ms - 200 threads - Peak TPS = 2790, Avg. TPS = 1716, mean rt = 87.33 ms - 400 threads - Peak TPS = 3810, Avg. TPS = 1800, mean rt = 165ms - 600 threads - Peak TPS = 4506.67, Avg. TPS = 1676.67, mean rt = 287.33ms 4 nodes Gluster (2 distribute of replicated 2 node) - 100 thread - peak TPS = 1293.33, Avg. TPS = 430, mean rt = 207.33ms - 200 threads - Peak TPS = 974.67, Avg. TPS = 245.33, mean rt = 672.67ms - 300 threads - Peak TPS = 861.33, Avg. TPS = 210, mean rt = 931.33 (no 400-600 threads since we run out of client machine, sorry). gfsd is configured to use 32 thread of iothread as brick. gfs-client is configured to use io-cache->write-behind->readahead->distribute->replicate. io-cache cache-size is 256MB. I used patched Fuse downloaded from Gluster web-site (build through DKMS). As the result yield, it seems that Gluster performance worse with increasing no. of client. One observation is that the glusterfs process on client is taking about 100% of CPU during all the tests. glusterfsd is utilizing only 70-80% of CPUs during the test time. Note that system is Dual core. I also tried using modglusterfs and not using fuse at all to serve all the static files and conduct another test with Grinder. The result is about the same, 1000+ peak TPS with 2-400 avg. TPS. A problem arise in this test that each Apache prefork process used more about twice more memory and we need to lower number of httpd processes by about half. I tried disable EnableMMAP and it didn't help much. Adjusting readahead, write behind according to GlusterOptimization page didn't help much either. My question is, there seems to be bottleneck in this setup, but how can I track this? Note that, I didn't do any other optimization other than what said above. Are there any best practice configuration for using Apache to serve a bunch of small static files like this around? Regards, Somsak _______________________________________________ Gluster-users mailing list Gluster-users http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users _______________________________________________ Gluster-users mailing list Gluster-users http://gluster.org/cgi-bin/mailman/listinfo/gluster-users regards, -- Raghavendra G ------------------------------------------------------------------------ _______________________________________________ Gluster-users mailing list Gluster-users http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
PATCH: http://patches.gluster.com/patch/943 in master (protocol/client: fixed registration of saved_fds)
PATCH: http://patches.gluster.com/patch/943 in release-2.0 (protocol/client: fixed registration of saved_fds)
The last two patches were wrongly logged for this bug. They were supposed to be for 192 instead.
Created attachment 55 [details] gnome metadata file, believed to contain the bug-triggering errors
Created attachment 56 [details] PPM test image
Created attachment 57 [details] Xlib test program.