Bug 463497
| Summary: | gfs file locking is extremely slow ( gfs_controld vs cluster locking ) | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 5 | Reporter: | Everett Bennett <everett.bennett> | ||||||
| Component: | gfs-kmod | Assignee: | David Teigland <teigland> | ||||||
| Status: | CLOSED NOTABUG | QA Contact: | Cluster QE <mspqa-list> | ||||||
| Severity: | urgent | Docs Contact: | |||||||
| Priority: | medium | ||||||||
| Version: | 5.2 | CC: | adas, bstevens, edamato, grimme, hlawatschek, luke2261-2017, mwaite, nstraz, rpeterso, sghosh | ||||||
| Target Milestone: | --- | ||||||||
| Target Release: | --- | ||||||||
| Hardware: | x86_64 | ||||||||
| OS: | Linux | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2009-12-21 18:31:49 UTC | Type: | --- | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Attachments: |
|
||||||||
Created attachment 317506 [details]
Current cluster.conf file.
Current cluster.conf file.
This looks like they are running RHEL 5.1 version of the code. The 8 node cluster have a mixture of RHEL 5.1 and RHEL 5.2 nodes that were all yum'ed to the latest patch level: ibmblade01/root> find /RHEL52 -name '*gfs*' -print /RHEL52/Server/gfs2-utils-0.1.44-1.el5.x86_64.rpm /RHEL52/ClusterStorage/kmod-gfs2-xen-1.92-1.1.el5.x86_64.rpm /RHEL52/ClusterStorage/kmod-gfs-0.1.23-5.el5.x86_64.rpm /RHEL52/ClusterStorage/kmod-gfs2-1.92-1.1.el5.x86_64.rpm /RHEL52/ClusterStorage/kmod-gfs-xen-0.1.23-5.el5.x86_64.rpm /RHEL52/ClusterStorage/gfs-utils-0.1.17-1.el5.x86_64.rpm ibmblade01/root> rpm -qa|grep gfs|sort gfs2-utils-0.1.44-1.el5_2.1 gfs-utils-0.1.12-1.el5 kmod-gfs-0.1.19-7.el5 kmod-gfs2-1.52-1.16.el5 kmod-gfs2-xen-1.52-1.16.el5 kmod-gfs-xen-0.1.19-7.el5 ibmblade01/root> ibmall.sh "cat /etc/issue" ----ibmblade08---- ssh ibmblade08 "cat /etc/issue" Red Hat Enterprise Linux Server release 5.2 (Tikanga) Kernel \r on an \m ----ibmblade07---- ssh ibmblade07 "cat /etc/issue" Red Hat Enterprise Linux Server release 5.2 (Tikanga) Kernel \r on an \m ----ibmblade06---- ssh ibmblade06 "cat /etc/issue" Red Hat Enterprise Linux Server release 5.2 (Tikanga) Kernel \r on an \m ----ibmblade05---- ssh ibmblade05 "cat /etc/issue" Red Hat Enterprise Linux Server release 5.2 (Tikanga) Kernel \r on an \m ----ibmblade04---- ssh ibmblade04 "cat /etc/issue" Red Hat Enterprise Linux Server release 5.2 (Tikanga) Kernel \r on an \m ----ibmblade03---- ssh ibmblade03 "cat /etc/issue" Red Hat Enterprise Linux Server release 5.2 (Tikanga) Kernel \r on an \m ----ibmblade02---- ssh ibmblade02 "cat /etc/issue" Red Hat Enterprise Linux Server release 5.2 (Tikanga) Kernel \r on an \m ----ibmblade01---- ssh ibmblade01 "cat /etc/issue" Red Hat Enterprise Linux Server release 5.2 (Tikanga) Kernel \r on an \m ----ibmblade08---- ssh ibmblade08 "rpm -qa --last|grep gfs" kmod-gfs-xen-0.1.23-5.el5 Sat 20 Sep 2008 02:13:40 PM EDT kmod-gfs-0.1.23-5.el5 Sat 20 Sep 2008 02:13:15 PM EDT gfs2-utils-0.1.44-1.el5_2.1 Fri 19 Sep 2008 02:25:24 PM EDT kmod-gfs2-1.52-1.16.el5 Wed 27 Aug 2008 03:08:29 PM EDT kmod-gfs-0.1.19-7.el5 Wed 27 Aug 2008 03:08:21 PM EDT kmod-gfs-xen-0.1.19-7.el5 Wed 27 Aug 2008 03:08:17 PM EDT kmod-gfs2-xen-1.52-1.16.el5 Wed 27 Aug 2008 03:07:56 PM EDT gfs-utils-0.1.12-1.el5 Wed 27 Aug 2008 03:03:31 PM EDT gfs2-utils-0.1.38-1.el5 Wed 27 Aug 2008 03:03:01 PM EDT ----ibmblade07---- ssh ibmblade07 "rpm -qa --last|grep gfs" kmod-gfs-xen-0.1.23-5.el5 Fri 19 Sep 2008 09:58:28 PM EDT kmod-gfs-0.1.23-5.el5 Fri 19 Sep 2008 09:58:03 PM EDT gfs2-utils-0.1.44-1.el5_2.1 Fri 19 Sep 2008 03:29:53 PM EDT kmod-gfs2-1.52-1.16.el5 Wed 27 Aug 2008 02:42:32 PM EDT kmod-gfs-0.1.19-7.el5 Wed 27 Aug 2008 02:42:22 PM EDT kmod-gfs-xen-0.1.19-7.el5 Wed 27 Aug 2008 02:42:18 PM EDT kmod-gfs2-xen-1.52-1.16.el5 Wed 27 Aug 2008 02:41:52 PM EDT gfs-utils-0.1.12-1.el5 Wed 27 Aug 2008 02:37:12 PM EDT ----ibmblade06---- ssh ibmblade06 "rpm -qa --last|grep gfs" gfs2-utils-0.1.44-1.el5_2.1 Tue 22 Jul 2008 09:02:00 AM EDT kmod-gfs-xen-0.1.23-5.el5 Fri 30 May 2008 07:19:37 AM EDT kmod-gfs2-xen-1.92-1.1.el5 Fri 30 May 2008 07:19:10 AM EDT kmod-gfs-0.1.23-5.el5 Fri 30 May 2008 07:18:56 AM EDT kmod-gfs2-1.92-1.1.el5 Fri 30 May 2008 07:18:31 AM EDT gfs-utils-0.1.17-1.el5 Fri 30 May 2008 06:56:56 AM EDT ----ibmblade05---- ssh ibmblade05 "rpm -qa --last|grep gfs" gfs2-utils-0.1.44-1.el5_2.1 Tue 22 Jul 2008 08:59:04 AM EDT kmod-gfs-xen-0.1.23-5.el5 Thu 29 May 2008 10:41:44 PM EDT kmod-gfs2-xen-1.92-1.1.el5 Thu 29 May 2008 10:41:17 PM EDT kmod-gfs-0.1.23-5.el5 Thu 29 May 2008 10:41:03 PM EDT kmod-gfs2-1.92-1.1.el5 Thu 29 May 2008 10:40:34 PM EDT gfs-utils-0.1.17-1.el5 Thu 29 May 2008 10:21:56 PM EDT ----ibmblade04---- ssh ibmblade04 "rpm -qa --last|grep gfs" gfs2-utils-0.1.44-1.el5_2.1 Thu 28 Aug 2008 03:49:43 PM EDT kmod-gfs2-1.52-1.16.el5 Sat 17 May 2008 11:48:19 PM EDT kmod-gfs-0.1.19-7.el5 Sat 17 May 2008 11:48:15 PM EDT kmod-gfs-xen-0.1.19-7.el5 Sat 17 May 2008 11:48:11 PM EDT kmod-gfs2-xen-1.52-1.16.el5 Sat 17 May 2008 11:47:55 PM EDT gfs-utils-0.1.12-1.el5 Sat 17 May 2008 11:42:52 PM EDT ----ibmblade03---- ssh ibmblade03 "rpm -qa --last|grep gfs" gfs2-utils-0.1.44-1.el5_2.1 Fri 19 Sep 2008 03:00:42 PM EDT kmod-gfs2-1.52-1.16.el5 Sat 17 May 2008 11:33:51 PM EDT kmod-gfs-0.1.19-7.el5 Sat 17 May 2008 11:33:47 PM EDT kmod-gfs-xen-0.1.19-7.el5 Sat 17 May 2008 11:33:43 PM EDT kmod-gfs2-xen-1.52-1.16.el5 Sat 17 May 2008 11:33:31 PM EDT gfs-utils-0.1.12-1.el5 Sat 17 May 2008 11:28:10 PM EDT ----ibmblade02---- ssh ibmblade02 "rpm -qa --last|grep gfs" gfs2-utils-0.1.44-1.el5_2.1 Fri 19 Sep 2008 03:00:28 PM EDT kmod-gfs2-1.52-1.16.el5 Sat 17 May 2008 11:18:07 PM EDT kmod-gfs-0.1.19-7.el5 Sat 17 May 2008 11:18:03 PM EDT kmod-gfs-xen-0.1.19-7.el5 Sat 17 May 2008 11:17:59 PM EDT kmod-gfs2-xen-1.52-1.16.el5 Sat 17 May 2008 11:17:43 PM EDT gfs-utils-0.1.12-1.el5 Sat 17 May 2008 11:12:26 PM EDT ----ibmblade01---- ssh ibmblade01 "rpm -qa --last|grep gfs" gfs2-utils-0.1.44-1.el5_2.1 Fri 19 Sep 2008 03:07:21 PM EDT kmod-gfs2-1.52-1.16.el5 Sat 17 May 2008 11:04:34 PM EDT kmod-gfs-0.1.19-7.el5 Sat 17 May 2008 11:04:29 PM EDT kmod-gfs-xen-0.1.19-7.el5 Sat 17 May 2008 11:04:26 PM EDT kmod-gfs2-xen-1.52-1.16.el5 Sat 17 May 2008 11:04:09 PM EDT gfs-utils-0.1.12-1.el5 Sat 17 May 2008 10:58:34 PM EDT The plock_ownership optimization was added in RHEL5.2 (cman package). When using that option, 5.1 and 5.2 are incompatible, so all nodes must be running the 5.2 cman package in order to set plock_ownership=1. I believe the necessary RHEL5.2 cman package version is cman-2.0.84-2.el5 Looks to me like ibmblade01-04 are running rhel 5.1 version of kmod-gfs, and ibmblade05-08 are running 5.2 versions. kmod-gfs-0.1.19-7.el5 - is RHEL 5.1 released version kmod-gfs-0.1.23-5.el5 - is RHEL 5.2 released version Question would be whether the rest of the cluster packages are at the same version level as the kmod-gfs rpms. I am having trouble upgrading a 4-node HP cluster in South Africa recently registered. cel201p> uname -a Linux cel201p 2.6.18-92.el5 #1 SMP Tue Apr 29 13:16:15 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux cel201p> cat /etc/issue Red Hat Enterprise Linux Server release 5.2 (Tikanga) Kernel \r on an \m yum fails with this error: Warning: rpmts_HdrFromFdno: Header V3 DSA signature: NOKEY, key ID 37017186 Importing GPG key 0x37017186 "Red Hat, Inc. (release key) <security>" from /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release Is this ok [y/N]: y Running rpm_check_debug Running Transaction Test Finished Transaction Test Transaction Check Error: file /etc/depmod.d/gfs2.conf from install of kmod-gfs2-1.92-1.1.el5_2.2 conflicts with file from package kmod-gfs2-1.52-1.16.el5 It also appears that when you try to "rpm" the package manually, it hangs. Here are the gfs packages on these nodes. Is there a safe way to upgrade these systems without breaking the kernel? cel201p> grep gfs /tmp/yum.installed.cel201.txt gfs2-utils.x86_64 0.1.38-1.el5 installed gfs2-utils.x86_64 0.1.44-1.el5_2.1 installed gfs-utils.x86_64 0.1.12-1.el5 installed gfs-utils.x86_64 0.1.17-1.el5 installed kmod-gfs2.x86_64 1.52-1.16.el5 installed kmod-gfs2.x86_64 1.92-1.1.el5_2.2 installed kmod-gfs.x86_64 0.1.19-7.el5 installed kmod-gfs.x86_64 0.1.23-5.el5_2.2 installed cel201p> grep gfs /tmp/yum.installed.cel204.txt gfs2-utils.x86_64 0.1.38-1.el5 installed gfs-utils.x86_64 0.1.12-1.el5 installed kmod-gfs2.x86_64 1.52-1.16.el5 installed kmod-gfs.x86_64 0.1.19-7.el5 installed cel201p> grep gfs /tmp/yum.installed.cel205.txt gfs2-utils.x86_64 0.1.38-1.el5 installed gfs-utils.x86_64 0.1.12-1.el5 installed kmod-gfs2.x86_64 1.52-1.16.el5 installed kmod-gfs.x86_64 0.1.19-7.el5 installed cel201p> grep gfs /tmp/yum.installed.cel206.txt gfs2-utils.x86_64 0.1.38-1.el5 installed gfs-utils.x86_64 0.1.12-1.el5 installed kmod-gfs2.x86_64 1.52-1.16.el5 installed kmod-gfs.x86_64 0.1.19-7.el5 installed Apparently, the kernel-2.6.18-92.1.13.el5.x86_64.rpm kernel is mismatched with the gfs related rpm's. (In reply to comment #7) > I am having trouble upgrading a 4-node HP cluster in South Africa recently > registered. ... > Transaction Check Error: > file /etc/depmod.d/gfs2.conf from install of kmod-gfs2-1.92-1.1.el5_2.2 > conflicts with file from package kmod-gfs2-1.52-1.16.el5 Go ahead and remove kmod-gfs2. That package is an update to the gfs2 kernel module which is also in the kernel package. You do not need it installed. This does not look good. cel204p> rpm -ev kmod-gfs2 WARNING: No module qla2xxx_conf found for kernel 2.6.18-92.1.13.el5, continuing anyway WARNING: No module qla2300 found for kernel 2.6.18-92.1.13.el5, continuing anyway WARNING: No module qla2400 found for kernel 2.6.18-92.1.13.el5, continuing anyway Stock kernel: cel206p> uname -a Linux cel206p 2.6.18-92.el5 #1 SMP Tue Apr 29 13:16:15 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux cel206p> lsmod|grep qla qla2400 242944 0 qla2300 159360 0 qla2xxx 319168 242 qla2400,qla2300 qla2xxx_conf 334856 1 intermodule 37508 2 qla2xxx,qla2xxx_conf scsi_mod 188665 10 mptctl,ib_iser,libiscsi,scsi_transport_iscsi,ib_srp,sg,qla2xxx,usb_storage,cciss,sd_mod New Kernel: cel201p> uname -a Linux cel201p 2.6.18-92.1.13.el5 #1 SMP Thu Sep 4 03:51:21 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux You have new mail in /var/spool/mail/root cel201p> lsmod|grep qla qla2xxx 1080481 154 scsi_transport_fc 73801 1 qla2xxx scsi_mod 188665 11 mptctl,ib_iser,libiscsi,scsi_transport_iscsi,ib_srp,sg,qla2xxx,scsi_transport_fc,usb_storage,cciss,sd_mod I finally was able to "yum" the South Africa HP 4-node Blade Cluster to the latest RHEL 5.2 to fix the gfs locking issue. I am booting now.... There was a conflict with the kernel patches, etc. I now have "2.6.18-92.1.13" installed on all 4 nodes. The upgrade did not go smoothly at all. Please review and be available if the cluster is not sane after boot. Turned off multipath daemon on all nodes. Now working. We have become aware of an new issue regarding gfs on a 4-node HP RHEL 5.2 cluster. We are unable to create a file/directory in the gfs mountpoint: cel201p> id uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel) cel201p> uname -a Linux cel201p 2.6.18-92.1.13.el5 #1 SMP Thu Sep 4 03:51:21 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux cel201p> cp /etc/motd /dcs/data01/motd.txt cp: cannot create regular file `/dcs/data01/motd.txt': Invalid argument cel201p> echo $? 1 cel201p> cp /etc/motd /dcs/data01/TEST/motd.txt cp: cannot create regular file `/dcs/data01/TEST/motd.txt': Invalid argument cel201p> echo $? 1 cel201p> cp /etc/motd /dcs/data01/DATA60/motd.txt cel201p> ls -ld /dcs/data01/motd.txt /dcs/data01/TEST/motd.txt /dcs/data01/DATA60/motd.txt -rw-r--r-- 1 root root 223 Oct 14 22:07 /dcs/data01/DATA60/motd.txt -rw-r--r-- 1 root root 0 Oct 14 22:06 /dcs/data01/motd.txt -rw-r--r-- 1 root root 0 Oct 14 22:06 /dcs/data01/TEST/motd.txt cel201p> service clvmd status clvmd (pid 31637) is running... active volumes: appl01lv appl02lv data01lv data02lv data03lv data04lv data05lv data06lv data07lv data08lv data09lv data10lv data11lv data12lv data13lv data14lv data15lv data16lv data17lv cel201p> lvs LV VG Attr LSize Origin Snap% Move Log Copy% Convert appl01lv appl01 -wi-ao 350.00G appl02lv appl02 -wi-ao 350.00G data01lv data01 -wi-ao 1000.00G data02lv data02 -wi-ao 1000.00G data03lv data03 -wi-ao 999.98G data04lv data04 -wi-ao 999.98G data05lv data05 -wi-ao 999.98G data06lv data06 -wi-ao 999.98G data07lv data07 -wi-ao 999.98G data08lv data08 -wi-ao 999.98G data09lv data09 -wi-ao 999.98G data10lv data10 -wi-ao 999.98G data11lv data11 -wi-ao 999.98G data12lv data12 -wi-ao 999.98G data13lv data13 -wi-ao 999.98G data14lv data14 -wi-ao 999.98G data15lv data15 -wi-ao 999.98G data16lv data16 -wi-ao 999.98G data17lv data17 -wi-ao 999.98G cel201p> mount|grep gfs none on /sys/kernel/config type configfs (rw) /dev/mapper/appl01-appl01lv on /dcs/appl01 type gfs (rw,hostdata=jid=0:id=131076:first=0) /dev/mapper/appl02-appl02lv on /dcs/appl02 type gfs (rw,hostdata=jid=0:id=262148:first=0) /dev/mapper/data01-data01lv on /dcs/data01 type gfs (rw,hostdata=jid=0:id=393220:first=0) /dev/mapper/data02-data02lv on /dcs/data02 type gfs (rw,hostdata=jid=0:id=524292:first=0) /dev/mapper/data03-data03lv on /dcs/data03 type gfs (rw,hostdata=jid=0:id=655364:first=0) /dev/mapper/data04-data04lv on /dcs/data04 type gfs (rw,hostdata=jid=0:id=786436:first=0) /dev/mapper/data05-data05lv on /dcs/data05 type gfs (rw,hostdata=jid=0:id=917508:first=0) /dev/mapper/data06-data06lv on /dcs/data06 type gfs (rw,hostdata=jid=0:id=1048580:first=0) /dev/mapper/data07-data07lv on /dcs/data07 type gfs (rw,hostdata=jid=0:id=1179652:first=0) /dev/mapper/data08-data08lv on /dcs/data08 type gfs (rw,hostdata=jid=0:id=1310724:first=0) /dev/mapper/data09-data09lv on /dcs/data09 type gfs (rw,hostdata=jid=0:id=1441796:first=0) /dev/mapper/data10-data10lv on /dcs/data10 type gfs (rw,hostdata=jid=0:id=1572868:first=0) /dev/mapper/data11-data11lv on /dcs/data11 type gfs (rw,hostdata=jid=0:id=1703940:first=0) /dev/mapper/data12-data12lv on /dcs/data12 type gfs (rw,hostdata=jid=0:id=1835012:first=0) /dev/mapper/data13-data13lv on /dcs/data13 type gfs (rw,hostdata=jid=0:id=1966084:first=0) /dev/mapper/data14-data14lv on /dcs/data14 type gfs (rw,hostdata=jid=0:id=2097156:first=0) /dev/mapper/data15-data15lv on /dcs/data15 type gfs (rw,hostdata=jid=0:id=2228228:first=0) /dev/mapper/data16-data16lv on /dcs/data16 type gfs (rw,hostdata=jid=0:id=2359300:first=0) /dev/mapper/data17-data17lv on /dcs/data17 type gfs (rw,hostdata=jid=0:id=2490372:first=0) I expect you'll need to raise a separate issue/bug to have the new problem looked at properly. The only suggestion I have at the moment is to look at 'gfs_tool df'. After updating to 5.2, are you still experiencing the same performance issues? I observed the same issue with 5.2 It took about 8 secs on a gfs filesystem to acquire and release 10 fcntl locks. An a local filesystem the test needed only 0.016 secs to finish. -Mark This has become a critical issue for one of our customers. Has there been any progess on this issue. Hi Gary, Don't know where to turn for help. Need advice. We were hit again with gfs locks just days after we did our hse keeping. Please see the attached. Am flabergasted with the situation and not able to explained why the gfs locks went so high 500K+ at 3:15pm today. Our guy did a routine check just before they went for lunch. And it was OK on number of gfs locks < 20k, and when they came back at 3:20pm, the gfs locks was already 500k+ and caused ImE to hanged! Is there any tools that we can use to tell us on gfs log, what and where it went wrong ?? Which processing is consuming high locks, etc? Before we shutdown the Blade and ImE Date 17th Apr and 3:15pm ssh cel201 /SYSTEM/APP/gfs_lock.sh dcs/data01a locks held 64994 dcs/data01b locks held 14431 dcs/data01c locks held 15689 dcs/data01d locks held 14134 dcs/data01e locks held 15091 dcs/data01f locks held 14390 dcs/data01g locks held 14325 dcs/data01h locks held 14806 dcs/data02a locks held 14867 dcs/data02b locks held 15189 dcs/data02c locks held 15304 dcs/data02d locks held 14286 dcs/data02e locks held 14505 dcs/data02f locks held 15580 dcs/data02g locks held 14817 ssh cel204 /SYSTEM/APP/gfs_lock.sh dcs/data01a locks held 1053 dcs/data01b locks held 17373 dcs/data01c locks held 17367 dcs/data01d locks held 16416 dcs/data01e locks held 17433 dcs/data01f locks held 16967 dcs/data01g locks held 16953 dcs/data01h locks held 17084 dcs/data02a locks held 524238 dcs/data02b locks held 16748 dcs/data02c locks held 17131 dcs/data02d locks held 17288 dcs/data02e locks held 17376 dcs/data02f locks held 17680 dcs/data02g locks held 17563 ssh cel205 /SYSTEM/APP/gfs_lock.sh dcs/data01a locks held 71794 dcs/data01b locks held 13681 dcs/data01c locks held 13736 dcs/data01d locks held 13157 dcs/data01e locks held 13758 dcs/data01f locks held 13249 dcs/data01g locks held 13566 dcs/data01h locks held 13906 dcs/data02a locks held 13274 dcs/data02b locks held 13181 dcs/data02c locks held 13542 dcs/data02d locks held 13723 dcs/data02e locks held 13430 dcs/data02f locks held 14588 dcs/data02g locks held 13968 ssh cel206 /SYSTEM/APP/gfs_lock.sh dcs/data01a locks held 72804 dcs/data01b locks held 15082 dcs/data01c locks held 17004 dcs/data01d locks held 14692 dcs/data01e locks held 14929 dcs/data01f locks held 15958 dcs/data01g locks held 15606 dcs/data01h locks held 16031 dcs/data02a locks held 14464 dcs/data02b locks held 14497 Regards Danny More information: GFS: fsid=Ime_cel_clu:data01f.1: jid=2: Trying to acquire journal lock... GFS: fsid=Ime_cel_clu:data01g.2: jid=1: Trying to acquire journal lock... GFS: fsid=Ime_cel_clu:data02e.3: jid=1: Trying to acquire journal lock... GFS: fsid=Ime_cel_clu:data02f.3: jid=2: Trying to acquire journal lock... GFS: fsid=Ime_cel_clu:data02g.3: jid=2: Trying to acquire journal lock... GFS: fsid=Ime_cel_clu:data02f.3: jid=2: Busy GFS: fsid=Ime_cel_clu:data08.3: jid=1: Busy GFS: fsid=Ime_cel_clu:appl02.3: jid=1: Busy GFS: fsid=Ime_cel_clu:data01b.2: jid=1: Busy GFS: fsid=Ime_cel_clu:data01a.1: jid=2: Busy GFS: fsid=Ime_cel_clu:data01e.1: jid=3: Busy GFS: fsid=Ime_cel_clu:data02e.3: jid=1: Busy GFS: fsid=Ime_cel_clu:data01h.1: jid=2: Busy GFS: fsid=Ime_cel_clu:data01f.1: jid=2: Busy GFS: fsid=Ime_cel_clu:data01g.2: jid=1: Busy GFS: fsid=Ime_cel_clu:data01c.1: jid=3: Busy GFS: fsid=Ime_cel_clu:appl01.3: jid=1: Busy GFS: fsid=Ime_cel_clu:data02b.3: jid=2: Busy GFS: fsid=Ime_cel_clu:data16.2: jid=1: Busy GFS: fsid=Ime_cel_clu:data14.1: jid=3: Busy GFS: fsid=Ime_cel_clu:data03.1: jid=3: Busy GFS: fsid=Ime_cel_clu:data09.3: jid=2: Busy GFS: fsid=Ime_cel_clu:data05.1: jid=3: Busy GFS: fsid=Ime_cel_clu:data13.3: jid=1: Busy GFS: fsid=Ime_cel_clu:data02g.3: jid=2: Busy GFS: fsid=Ime_cel_clu:data10.3: jid=1: Busy GFS: fsid=Ime_cel_clu:data06.1: jid=3: Busy GFS: fsid=Ime_cel_clu:data02a.3: jid=1: Busy GFS: fsid=Ime_cel_clu:data01d.2: jid=3: Busy GFS: fsid=Ime_cel_clu:data02d.3: jid=1: Busy GFS: fsid=Ime_cel_clu:data02c.3: jid=1: Busy GFS: fsid=Ime_cel_clu:data12.2: jid=1: Busy GFS: fsid=Ime_cel_clu:data07.2: jid=1: Busy GFS: fsid=Ime_cel_clu:data04.1: jid=3: Busy GFS: fsid=Ime_cel_clu:data11.1: jid=2: Busy GFS: fsid=Ime_cel_clu:data15.3: jid=2: Looking at journal... GFS: fsid=Ime_cel_clu:data17.3: jid=1: Busy GFS: fsid=Ime_cel_clu:data15.3: jid=2: Acquiring the transaction lock... GFS: fsid=Ime_cel_clu:data15.3: jid=2: Replaying journal... GFS: fsid=Ime_cel_clu:data15.3: jid=2: Replayed 0 of 1 blocks GFS: fsid=Ime_cel_clu:data15.3: jid=2: replays = 0, skips = 0, sames = 1 GFS: fsid=Ime_cel_clu:data15.3: jid=2: Journal replayed in 1s GFS: fsid=Ime_cel_clu:data15.3: jid=2: Done cel201p> gfs_tool df /dcs/data15 /dcs/data15: SB lock proto = "lock_dlm" SB lock table = "Ime_cel_clu:data15" SB ondisk format = 1309 SB multihost format = 1401 Block size = 4096 Journals = 8 Resource Groups = 3992 Mounted lock proto = "lock_dlm" Mounted lock table = "Ime_cel_clu:data15" Mounted host data = "jid=3:id=3080193:first=0" Journal number = 3 Lock module flags = 0 Local flocks = FALSE Local caching = FALSE Oopses OK = FALSE Type Total Used Free use% ------------------------------------------------------------------------ inodes 1721675 1721675 0 100% metadata 664793 312132 352661 47% data 259188688 132142484 127046204 51% cel201p> gfs_tool gettune /dcs/data15 ilimit1 = 100 ilimit1_tries = 3 ilimit1_min = 1 ilimit2 = 500 ilimit2_tries = 10 ilimit2_min = 3 demote_secs = 300 incore_log_blocks = 1024 jindex_refresh_secs = 60 depend_secs = 60 scand_secs = 5 recoverd_secs = 60 logd_secs = 1 quotad_secs = 5 inoded_secs = 15 glock_purge = 0 quota_simul_sync = 64 quota_warn_period = 10 atime_quantum = 3600 quota_quantum = 60 quota_scale = 1.0000 (1, 1) quota_enforce = 1 quota_account = 1 new_files_jdata = 0 new_files_directio = 0 max_atomic_write = 4194304 max_readahead = 262144 lockdump_size = 131072 stall_secs = 600 complain_secs = 10 reclaim_limit = 5000 entries_per_readdir = 32 prefetch_secs = 10 statfs_slots = 64 max_mhc = 10000 greedy_default = 100 greedy_quantum = 25 greedy_max = 250 rgrp_try_threshold = 100 statfs_fast = 0 More info:
cel201p> gfs_tool counters /dcs/data15
locks 4747
locks held 696
freeze count 0
incore inodes 375
metadata buffers 131
unlinked inodes 0
quota IDs 0
incore log buffers 0
log space used 0.10%
meta header cache entries 0
glock dependencies 0
glocks on reclaim list 0
log wraps 0
outstanding LM calls 0
outstanding BIO calls 0
fh2dentry misses 0
glocks reclaimed 3257294
glock nq calls 4490668
glock dq calls 4195618
glock prefetch calls 51
lm_lock calls 3555977
lm_unlock calls 3258253
lm callbacks 6817944
address operations 6551
dentry operations 16680
export operations 0
file operations 5721
inode operations 245091
super operations 958858
vm operations 0
block I/O reads 0
block I/O writes 0
Right now, the file system /dcs/data01a seems to have the highest number of
cel204p> gfs_tool df /dcs/data01a
/dcs/data01a:
SB lock proto = "lock_dlm"
SB lock table = "Ime_cel_clu:data01a"
SB ondisk format = 1309
SB multihost format = 1401
Block size = 4096
Journals = 8
Resource Groups = 976
Mounted lock proto = "lock_dlm"
Mounted lock table = "Ime_cel_clu:data01a"
Mounted host data = "jid=0:id=458753:first=1"
Journal number = 0
Lock module flags = 0
Local flocks = FALSE
Local caching = FALSE
Oopses OK = FALSE
Type Total Used Free use%
------------------------------------------------------------------------
inodes 4370443 4370443 0 100%
metadata 269010 145890 123120 54%
data 59316347 49531315 9785032 84%
cel204p> gfs_tool counters /dcs/data01a
locks 1024331
locks held 518195
freeze count 0
incore inodes 510454
metadata buffers 7768
unlinked inodes 89
quota IDs 2
incore log buffers 1
log space used 40.67%
meta header cache entries 269
glock dependencies 223
glocks on reclaim list 0
log wraps 48
outstanding LM calls 0
outstanding BIO calls 0
fh2dentry misses 0
glocks reclaimed 2179760
glock nq calls 12548550
glock dq calls 11420595
glock prefetch calls 1347710
lm_lock calls 2972967
lm_unlock calls 1703035
lm callbacks 4723864
address operations 1824653
dentry operations 559559
export operations 2074898
file operations 913127
inode operations 5545237
super operations 7193676
vm operations 0
block I/O reads 0
block I/O writes 0
There is no glock_purge-ing enabled. Try to use gfs_tool settune /dcs/data15 glockpurge 50 to purge every scand_secs glockpurge <50> % of unused glocks. That should increase the performance. OK By the way... The cluster was upgraded to RHEL 5.2 since Nov 13 2008. cel204p> cat /etc/issue Red Hat Enterprise Linux Server release 5.2 (Tikanga) Kernel \r on an \m cel204p> uname -a Linux cel204p 2.6.18-92.1.13.el5 #1 SMP Thu Sep 4 03:51:21 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux Is there any risk to applying this tuneable to all gfs file systems at the same time? with this "tuneable" we never had any problems. I am not sure why the folks in Malaysia, not South Africa, unable to get this kind of information from RedHat there?
So, I applied to all nodes:
cel204p> cat /SYSTEM/APP/gfs_set_purge.sh
grep gfs /etc/fstab|awk '{print $2}'|sort|while read args; do gfs_tool settune ${args} glock_purge 50; done
ssh cel201 "/SYSTEM/APP/gfs_lock.sh"
dcs/data01a locks held 30
dcs/data01b locks held 2070
dcs/data01c locks held 1964
dcs/data01d locks held 1922
dcs/data01e locks held 1986
dcs/data01f locks held 1758
dcs/data01g locks held 2090
dcs/data01h locks held 2052
dcs/data02a locks held 2123
dcs/data02b locks held 1202
dcs/data02c locks held 1857
dcs/data02d locks held 1193
dcs/data02e locks held 2030
dcs/data02f locks held 2350
dcs/data02g locks held 1980
----cel204----
ssh cel204 "/SYSTEM/APP/gfs_lock.sh"
dcs/data01a locks held 916
dcs/data01b locks held 676
dcs/data01c locks held 650
dcs/data01d locks held 679
dcs/data01e locks held 687
dcs/data01f locks held 684
dcs/data01g locks held 650
dcs/data01h locks held 752
dcs/data02a locks held 702
dcs/data02b locks held 654
dcs/data02c locks held 710
dcs/data02d locks held 715
dcs/data02e locks held 684
dcs/data02f locks held 863
dcs/data02g locks held 783
----cel205----
ssh cel205 "/SYSTEM/APP/gfs_lock.sh"
dcs/data01a locks held 34
dcs/data01b locks held 2184
dcs/data01c locks held 2181
dcs/data01d locks held 2535
dcs/data01e locks held 3285
dcs/data01f locks held 3093
dcs/data01g locks held 2357
dcs/data01h locks held 3285
dcs/data02a locks held 3155
dcs/data02b locks held 3345
dcs/data02c locks held 3232
dcs/data02d locks held 3168
dcs/data02e locks held 2502
dcs/data02f locks held 3417
dcs/data02g locks held 3326
----cel206----
ssh cel206 "/SYSTEM/APP/gfs_lock.sh"
dcs/data01a locks held 48
dcs/data01b locks held 1099
dcs/data01c locks held 974
dcs/data01d locks held 1213
dcs/data01e locks held 1909
dcs/data01f locks held 1795
dcs/data01g locks held 1931
dcs/data01h locks held 1823
dcs/data02a locks held 1260
dcs/data02b locks held 1265
dcs/data02c locks held 1100
dcs/data02d locks held 1298
dcs/data02e locks held 2024
dcs/data02f locks held 2238
dcs/data02g locks held 2073
cel204p>
Can you remind me if the "settune" is permanment, or do I need to place in a RC script on system boot? Never mind, I found this article. http://kbase.redhat.com/faq/docs/DOC-6533;jsessionid=760B8BA00F64E8D201107C857BE4EC7E.ab46478d My experience is that our application typically has issues similar to Oracle, do you have "known" parameters recommended for situations where Oracle my run on top of GFS? This bugzilla is a circus, a dumping ground for all sorts of completely unrelated issues over the past nine months. Someone please close this and open proper bugs for any oustanding problems. No. They are not unrelated issues. Do what you got to do? But, this problem is not resolved until we know it is working. I suggest that maybe you read the entire ticket and respond according. This ticket was submitted 09/23/2008! I see at least 3 completely unrelated problems in this bz: 1. posix lock performance in gfs_controld 2. "We are unable to create a file/directory in the gfs mountpoint" 3. excessive glock caching OK, as far as I am concerned, the original problem remains the same. The stuff in the middle was related to upgrading to the "current" RHEL 5.2 level at that time. The problems existed even after the upgrade of the OS. The customer has been experimenting with rebooting and "stopping" the application periodically to "free" the locks, etc. This was the solution that the customer obtained from the "local" RedHat support. Go figure! It is now simply a tuning issue, that will be great. But, this is the problem and always has been the problem. Created an attachment (id=317505) [details] wire_test.c Description of problem: We have had an historical issue with gfs and file locking. I'll do some testing with the scripts mentioned above to see what affect these parameters have? Everett - This bz has been stagnant for quite a while and as earlier comments indicate it seems to be a bunch of issues all mixed up together. I'm going to close this for the time being, and if you have specific issues can you please open up additional bugzillas for each specific issue so we can track them individually? Thanks |
Created attachment 317505 [details] wire_test.c Description of problem: We have had an historical issue with gfs and file locking. We have been testing gfs performance and have found that gfs locking (fcntl style) is extremely slow. The test consists of randomly reading/writing 512 byte blocks inside a file of roughly 25MB in size. Prior to each read/write, a 1 byte lock is placed on byte zero using code like this: flock_buf.l_type = F_UNLCK; flock_buf.l_whence = SEEK_SET; flock_buf.l_start = (off_t) 0; flock_buf.l_len = 1; sts = fcntl(fd, F_SETLKW, &flock_buf); The test runs 500-1,000 slower when locking is done vs. when it is not. Sample code (write_test.c) is attached. Compile as follows: gcc -g src/write_test.c -o bin/write_test The command line to run is: write_test -size 512 -max 50000 -reads 1000000 -rdwr -lock -create -file /gfs-fs/wtest Remove the -lock option to run without locks to see the vast difference. For comparison sake, we ran write_test on some other file systems, on a Xeon 3Ghz machine (seconds to complete 1M writes): With Locks Without Locks IBM gpfs 3.606 secs 2.622 ext3 3.685 2.624 gfs > 1hr 9.312 Version-Release number of selected component (if applicable): mod-gfs2-1.52-1.16.el5 gfs-utils-0.1.12-1.el5 kmod-gfs-0.1.19-7.el5 gfs2-utils-0.1.44-1.el5_2.1 How reproducible: # Without locking. write_test -create -size 512 -max 500000 -rdwr -reads 1000000 -file FILENAME # With locking. write_test -create -size 512 -max 500000 -rdwr -reads 1000000 -file FILENAME -lock Actual results: ibmblade01/root> echo gfs test gfs test --- ibmblade01/root> write_test -create -size 512 -max 500000 -rdwr -reads 1000000 -file /dcs/appl03/tmp/wtest Start Create Created 500000 records in 3.23959 secs @ 154341 rec/sec Start Read Read 1000000 records in 9.42082 secs @ 106148 rec/sec ibmblade01/root> write_test -create -size 512 -max 500000 -rdwr -reads 1000000 -file /dcs/appl03/tmp/wtest -lock Start Create ***** Takes 10 minutes for the create lock test to complete....and the read test that follows....never comes back. ***** Expected results: ibmblade01/root> echo ext3 test ext3 test --- ibmblade01/root> write_test -create -size 512 -max 500000 -rdwr -reads 1000000 -file /dcs/appl02_loc/tmp/wtest Start Create Created 500000 records in 1.12599 secs @ 444054 rec/sec Start Read Read 1000000 records in 3.56083 secs @ 280833 rec/sec ibmblade01/root> write_test -create -size 512 -max 500000 -rdwr -reads 1000000 -file /dcs/appl02_loc/tmp/wtest -lock Start Create Created 500000 records in 1.63628 secs @ 305571 rec/sec Start Read Read 1000000 records in 4.48847 secs @ 222793 rec/sec Additional info: ibmblade02/root> uname -a Linux ibmblade02 2.6.18-92.1.10.el5 #1 SMP Wed Jul 23 03:56:11 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux ibmblade02/root> cat /etc/issue Red Hat Enterprise Linux Server release 5.2 (Tikanga) Kernel \r on an \m We have this entry defined in cluster.conf: </rm> <gfs_controld plock_ownership="1" plock_rate_limit="0"/> </cluster> We have this defined in /etc/lvm/lvm.conf: ibmblade01/root> grep locking /etc/lvm/lvm.conf locking_type = 3