Description of problem: As per discussion in BZ#1370090, a feature request has been opened to extend the capability of gnfs process failover with CTDB. In event script, some modification are required for RHEL6 & RHEL7. Here is the script, 1 #!/bin/sh 2 # Event script to monitor GlusterFS NFS in a cluster environment 3 # Source: https://xrsa.net 4 # Author: Ben Draper 5 # Email: ben 6 # Install Location: /etc/ctdb/events.d/60.glusternfs 7 # 8 9 [ -n "$CTDB_BASE" ] || \ 10 export CTDB_BASE=$(cd -P $(dirname "$0") ; dirname "$PWD") 11 12 . $CTDB_BASE/functions 13 14 service_ports="111 2049 38465 38466" 15 16 # Placeholder 17 service_name="glusterfs_nfs" 18 19 # Verify GlusterFS Services and running 20 verify_supporting_services () 21 { 22 l_ret=0 23 /sbin/service glusterd status >/dev/null 2>&1 || { 24 echo "Service - glusterd is not running" 25 l_ret=1 26 } 27 28 /sbin/service glusterd status >/dev/null 2>&1 || { 29 echo "Service - glusterfsd is not running" 30 l_ret=1 31 } 32 33 /sbin/service rpcbind status >/dev/null 2>&1 || { 34 echo "Service - rpcbind is not running" 35 l_ret=1 36 } 37 38 if [ $l_ret -eq 1 ]; then 39 exit $l_ret 40 fi 41 } 42 43 # This will verify all required ports are listening 44 verify_ports () 45 { 46 l_ret=0 47 for check_port in $service_ports; do 48 ctdb_check_tcp_ports $check_port || { 49 l_ret=1 50 } 51 done 52 53 if [ $l_ret -eq 1 ]; then 54 exit $l_ret 55 fi 56 } 57 58 loadconfig 59 case "$1" in 60 monitor) 61 verify_supporting_services 62 verify_ports 63 update_tickles 2049 64 ;; 65 66 *) 67 ctdb_standard_event_handler "$@" 68 ;; 69 esac 70 71 exit 0 Additional info: - https://bugzilla.samba.org/show_bug.cgi?id=11238 which is reported in upstream for gnfs process failover - Require the doc text to be provided in the admin guide. Thanks Mukul
*** Bug 1294227 has been marked as a duplicate of this bug. ***
Mukul, Have you tried it along with CTDB_MANAGES_NFS=yes?
Surabhi, No, I haven't tested with "CTDB_MANAGES_NFS=yes" parameter. Mukul
(In reply to surabhi from comment #9) > Mukul, > > Have you tried it along with CTDB_MANAGES_NFS=yes? For RHEL 6 Red Hat Enterprise Linux Server release 6.8 (Santiago) Red Hat Gluster Storage Server 3.1 Update 3 Kernel Version : 2.6.32-642.11.1.el6.x86_64 Glusterfs-server-3.7.1-11.el6rhs.x86_64 ctdb-4.4.5-3.el6rhs.x86_64 Test 1 : Without CTDB_MANAGES_NFS=yes in /etc/sysconfig/ctdb ctdb doesnot recognise the failure of NFS server on another peer. There isn't any logs reported in /etc/sysconfig/ctdb for another node nfs server failure. Even ctdb status shows all nodes are in OK condition. Test 2 : With CTDB_MANAGES_NFS=yes failover of NFS server happens but with some issues CTDB_MANAGES_NFS=yes parameter enables kernel nfs everytime service get started,due to which ctdb reports error - [27647]: 60.nfs: ERROR: nfs failed RPC check: - [27647]: 60.nfs: rpcinfo: RPC: Program not registered so we need to stop the kernel nfs & restart the glusterd process on the server. Config File : [root@gluserver601 ~]# cat /etc/sysconfig/ctdb CTDB_RECOVERY_LOCK=/gluster/lock/lockfile CTDB_PUBLIC_ADDRESSES=/etc/ctdb/public_addresses CTDB_MANAGES_SAMBA=no CTDB_SAMBA_SKIP_SHARE_CHECK=yes CTDB_MANAGES_NFS=yes For RHEL 7 Red Hat Enterprise Linux Server release 7.3 (Maipo) Red Hat Gluster Storage Server 3.1 Update 3 Kernel Version : 3.10.0-514.2.2.el7.x86_643.10.0-514.2.2.el7.x86_64 Glusterfs-server-3.7.9-12.el7rhgs.x86_64 ctdb-4.4.5-3.el7rhgs.x86_64 Test 1 : Without CTDB_MANAGES_NFS=yes in /etc/sysconfig/ctdb, ctdb doesnot recognise the failure of NFS server on another peer. There isn't any logs reported in /etc/sysconfig/ctdb for another node nfs server failure. Even ctdb status shows all nodes are in OK condition. Test 2 : With CTDB_MANAGES_NFS=yes failover of NFS server happens without any issues. Here, in RHEL 7 Kernel nfs doesn't start as similar to RHEL6. Config File : [root@gluserver601 ~]# cat /etc/sysconfig/ctdb CTDB_RECOVERY_LOCK=/gluster/lock/lockfile CTDB_PUBLIC_ADDRESSES=/etc/ctdb/public_addresses CTDB_MANAGES_SAMBA=no CTDB_SAMBA_SKIP_SHARE_CHECK=yes CTDB_MANAGES_NFS=yes
Surabhi/Rtalur, Please have a look at comment#11. Thanks, Bipin Kunal
If ctdb version on both RHEL6 and RHEL7 were same, the above output could only be explained by not having having properly blacklisted kernel nfs module on RHEL6. Could you please recheck it once?
Hi, We tried blacklisting kernel nfs as per https://access.redhat.com/solutions/41278 but we got blank export list from both the gluster servers, though all the brick and gluster nfs processes are running fine on the servers. root@gluserver601 ~]# gluster v status nfs Status of volume: nfs Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick gluserver601:/brick01/b01 49153 0 Y 1785 Brick gluserver602:/brick01/b01 49153 0 Y 1652 NFS Server on localhost 2049 0 Y 1767 Self-heal Daemon on localhost N/A N/A Y 1775 NFS Server on gluserver602 2049 0 Y 1635 Self-heal Daemon on gluserver602 N/A N/A Y 1646 Task Status of Volume nfs ------------------------------------------------------------------------------ There are no active volume tasks [root@gluserver601 ~]# ctdb status Number of nodes:2 pnn:0 10.65.6.26 OK (THIS NODE) pnn:1 10.65.6.27 OK Generation:360803134 Size:2 hash:0 lmaster:0 hash:1 lmaster:1 Recovery mode:NORMAL (0) Recovery master:1 [root@gluserver601 ~]# showmount -e Export list for gluserver601: [root@gluserver601 ~]# cat /etc/modprobe.d/local-blacklist.conf blacklist nfsd install nfsd /bin/false [root@gluserver601 ~]# Regards, Abhishek Kumar
Did you restart the node after making blacklist changes? else you would have to do ensure that kernel nfs is completely shutoff in this session. I suspect kernel nfs is also running.
yes, i have rebooted the servers after blacklisting kernel nfs. but after the reboot when i check nfs process status,it came like below. [root@gluserver601 ~]#/etc/init.d/nfs status rpc.svcgssd is stopped rpc.mountd (pid 2998) is running... nfsd dead but subsys locked rpc.rquotad (pid 2993) is running... From the ctdb logs, i found below lines 2017/01/11 11:30:50.512364 [ 2084]: Running the "startup" event. 2017/01/11 11:30:50.839739 [ 2084]: 60.nfs: Shutting down NFS daemon: [FAILED] 2017/01/11 11:30:50.864499 [ 2084]: 60.nfs: Shutting down NFS mountd: [FAILED] 2017/01/11 11:30:50.885174 [ 2084]: 60.nfs: Shutting down NFS quotas: [FAILED] 2017/01/11 11:30:51.129483 [ 2084]: 60.nfs: Stopping NFS statd: [ OK ] 2017/01/11 11:30:51.210373 [ 2084]: 60.nfs: Starting NFS statd: [ OK ] 2017/01/11 11:30:51.597710 [ 2084]: 60.nfs: FATAL: Error running install command for nfsd 2017/01/11 11:30:51.602162 [ 2084]: 60.nfs: Starting NFS services: [ OK ] 2017/01/11 11:30:51.644990 [ 2084]: 60.nfs: Starting NFS quotas: [ OK ] 2017/01/11 11:30:51.707941 [ 2084]: 60.nfs: Starting NFS mountd: [ OK ] 2017/01/11 11:30:51.768292 [ 2084]: 60.nfs: Starting NFS daemon: rpc.nfsd: Unable to access /proc/fs/nfsd errno 2 (No such file or directory). 2017/01/11 11:30:51.768704 [ 2084]: 60.nfs: Please try, as root, 'mount -t nfsd nfsd /proc/fs/nfsd' and then restart rpc.nfsd to correct the problem 2017/01/11 11:30:51.780408 [ 2084]: 60.nfs: [FAILED] 2017/01/11 11:30:51.804062 [ 2084]: startup event OK - enabling monitoring 2017/01/11 11:30:51.804102 [ 2084]: Set runstate to RUNNING (5) NFS_HOSTNAME is not configured. statd-callout failed 2017/01/11 11:30:54.210867 [ 2084]: monitor event OK - node re-enabled 2017/01/11 11:30:54.210923 [ 2084]: Node became HEALTHY. Ask recovery master to reallocate IPs 2017/01/11 11:30:54.211428 [recoverd: 2200]: Node 0 has changed flags - now 0x0 was 0x2 2017/01/11 11:30:54.772869 [recoverd: 2200]: Disabling takeover runs for 60 seconds 2017/01/11 11:30:54.933699 [ 2084]: Takeover of IP 10.65.6.31/0 on interface eth0 2017/01/11 11:30:55.241990 [ 2084]: 60.nfs: Reconfiguring service "nfs"... If do service glusterd restart, showmount -e command start showing exported directory but even now, client is not able to mount the gluster volume To overcome this problem i have to kill the earlier started rpc.rquotad & rpc.mountd and restart the glusterd process. From the gluster nfs logs we found below error while mounting the volume [2017-01-11 06:11:19.209487] W [rpcsvc.c:278:rpcsvc_program_actor] 0-rpc-service: RPC program version not available (req 100003 4) for 10.65.8.127:819 [2017-01-11 06:11:19.209563] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully I suspect that rpc.rquotad & rpc.mountd started by ctdb doesn't work with glusterd and these services need to started by glusterd itself to work properly. After killing rpc.rquotad & rpc.mountd started eariler and restart of gluster process. [root@gluserver601 ~]# /etc/init.d/nfs status rpc.svcgssd is stopped rpc.mountd dead but subsys locked nfsd dead but subsys locked rpc.rquotad (pid 11109) is running... [root@gluserver601 ~]# So, is there a way to restrict ctdb to start/stop the nfs services and only glusterd should start/stop the nfs related services? Regards, Abhishek Kumar
Hi all, I'm showing up since I have a working setup in production on CentOS 7 with Gluster 3.7.17 My quick notes: *) I have CTDB_MANAGES_NFS=no (actually, it's commented out at all) since that basically means "use *kernel-based* NFS server" and we do not want it *) As a consequence of the above point, I do not need any kernel module blacklisting (which I think could be confusing: you ask for it in CTDB by setting the MANAGE variable to yes, then you deny its kernel module loading) *) I enable NFS by means of "gluster vol set nfs.disable off" on the Gluster volumes where it's needed *) I enable Samba shares by means of explicit smb.conf configuration (I keep user.cifs disabled on all volumes) *) I disable automatic management of Gluster CTDB-locking volume: I created a pair of systemd units: one for waiting for the custom shared-lock Gluster volume to come online and the other to actually FUSE-mount it; I modified CTDB systemd unit to depend/wait on these *) the only RPC service that I enable is rpcbind (everything else is disabled in systemd configuration) *) I use a modified version of the monitoring script in Description above: - line 20: verify_supporting_services: I use equivalent systemctl calls (since I'm on CentOS 7) on all checks inside this function (I use "systemctl -q is-active unitname") - line 28: you should apparently change glusterd in glusterfsd (otherwise it's the same check as on line 23), but I found that I need to comment out all lines from 28 up to 31 because glusterfsd is started by glusterd and not on its own (at least on my systemd-based CentOS 7) Everything works so well that I'm using a CTDB-based workaround to keep the cluster running in presence of frequent Gluster-NFS crashes (tracked in other Bugzilla entries) HTH
Created attachment 1241515 [details] /etc/ctdb/events.d/60.glusternfs modified/tested for CentOS 7
(In reply to Giuseppe Ragusa from comment #17) > Hi all, > I'm showing up since I have a working setup in production on CentOS 7 with > Gluster 3.7.17 > > My quick notes: > > *) I have > CTDB_MANAGES_NFS=no > (actually, it's commented out at all) since that basically means "use > *kernel-based* NFS server" and we do not want it > > *) As a consequence of the above point, I do not need any kernel module > blacklisting (which I think could be confusing: you ask for it in CTDB by > setting the MANAGE variable to yes, then you deny its kernel module loading) > > *) I enable NFS by means of "gluster vol set nfs.disable off" on the Gluster > volumes where it's needed > > *) I enable Samba shares by means of explicit smb.conf configuration (I keep > user.cifs disabled on all volumes) > > *) I disable automatic management of Gluster CTDB-locking volume: I created > a pair of systemd units: one for waiting for the custom shared-lock Gluster > volume to come online and the other to actually FUSE-mount it; I modified > CTDB systemd unit to depend/wait on these > > *) the only RPC service that I enable is rpcbind (everything else is > disabled in systemd configuration) > > *) I use a modified version of the monitoring script in Description above: > - line 20: verify_supporting_services: I use equivalent systemctl calls > (since I'm on CentOS 7) on all checks inside this function (I use "systemctl > -q is-active unitname") > - line 28: you should apparently change glusterd in glusterfsd (otherwise > it's the same check as on line 23), but I found that I need to comment out > all lines from 28 up to 31 because glusterfsd is started by glusterd and not > on its own (at least on my systemd-based CentOS 7) > > Everything works so well that I'm using a CTDB-based workaround to keep the > cluster running in presence of frequent Gluster-NFS crashes (tracked in > other Bugzilla entries) > > HTH Thanks Giuseppe. What you have done here is right thing to do. Hence the RFE flag on this bug. The proper fix is to write script specially for gluster-NFS and add it to ctdb package. It will have your changes and a little more. After that we can set ctdb_manages_nfs to yes and also point ctdb to new scripts using CTDB_NFS_CALLOUT variable. We are currently working on having a cli for starting/stopping nfs in glusterd. After that we will be able to change nfs callout scripts to monitor and manage gluster NFS services.
Hello Raghavendra, I have provided a work- around for RHEL6 failover but this requires manual intervention to work-properly. Steps are : 1) Insert CTDB_MANAGES_NFS=yes in /etc/sysconfig/ctdb 2) Black-list kernel-nfs create a new file as /etc/modprobe.d/local-blacklist.conf blacklist nfsd install nfsd /bin/false 3) Reboot the node 4) Check /etc/init.d/nfs status 5) Kill if any services found running. 6) Restart the glusterd service glusterd restart 7) Agin verify if rpc.mountd is running or not from /etc/init.d/nfs status So,Could we do something to stop rpc.mountd from getting started everytime when ctdb restart ?
I tried black-listing below modules. blacklist nfsd blacklist lockd blacklist nfs_acl blacklist auth_rpcgss blacklist sunrpc blacklist exportfs But none seem to be stopping rpc-mountd service from coming up. Request Steve/Ben to comment on if there is any way to block rpc.mountd from coming up (by disabling any kernel module or by any other means). Thanks!
On RHEL6, isn't rpc.mountd started by /etc/init.d/nfs? I would run `chkconfig nfs off` which will remove the init symlinks that would start it. That seems too simple though, what am I missing?
Sorry for the confusion. I think the use-case here is to prevent rpc-mountd.service from coming up when "service nfs start" is executed or kernel-nfs server is being started. To elaborate - * CTDB manages kernel-NFS server and tries to start that service when ever it restarts. We have a usecase to prevent it (this BZ RFE). * So as an interim solution/work-around, to prevent NFS server from coming up, it was suggested to blacklist nfsd kernel module. But that seem to be preventing only nfsd daemon but rpc.mountd service still manages to come up. So is there any way to disable rpc.mountd from being started when "service nfs start" is executed in short. @Abhishek, Could you please confirm if the details above are accurate.
What is needed from "service nfs start"? I imagine you could replace "service nfs start" with whatever is required, leaving out rpc.mountd.
* With the current version of ctdb startup script,whenever ctdb get started, it tries to start nfs and check whether these processes are running or not. Please refer below logs of ctdb 2017/01/26 08:20:55.678759 [10931]: Running the "startup" event. 2017/01/26 08:20:55.942137 [10931]: 60.nfs: Shutting down NFS daemon: [FAILED] 2017/01/26 08:20:55.962234 [10931]: 60.nfs: Shutting down NFS mountd: [FAILED] 2017/01/26 08:20:55.983852 [10931]: 60.nfs: Shutting down NFS quotas: [FAILED] 2017/01/26 08:20:56.084514 [10931]: 60.nfs: Stopping NFS statd: [FAILED] 2017/01/26 08:20:56.169333 [10931]: 60.nfs: Starting NFS statd: [ OK ] 2017/01/26 08:20:56.247628 [10931]: 60.nfs: WARNING: Error running install command for sunrpc 2017/01/26 08:20:56.257390 [10931]: 60.nfs: WARNING: Error inserting nfs_acl (/lib/modules/2.6.32-642.11.1.el6.x86_64/kernel/fs/nfs_common/nfs_acl.ko): Unknown symbol in module, or unknown parameter (see dmesg) 2017/01/26 08:20:56.257454 [10931]: 60.nfs: WARNING: Error inserting lockd (/lib/modules/2.6.32-642.11.1.el6.x86_64/kernel/fs/lockd/lockd.ko): Unknown symbol in module, or unknown parameter (see dmesg) 2017/01/26 08:20:56.257502 [10931]: 60.nfs: FATAL: Error inserting nfsd (/lib/modules/2.6.32-642.11.1.el6.x86_64/kernel/fs/nfsd/nfsd.ko): Unknown symbol in module, or unknown parameter (see dmesg) 2017/01/26 08:20:56.264679 [10931]: 60.nfs: Starting NFS services: [ OK ] 2017/01/26 08:20:56.272068 [10931]: 60.nfs: Starting NFS quotas: [ OK ] 2017/01/26 08:20:56.279822 [10931]: 60.nfs: Starting NFS mountd: [ OK ] 2017/01/26 08:20:56.339912 [10931]: 60.nfs: Starting NFS daemon: rpc.nfsd: Unable to access /proc/fs/nfsd errno 2 (No such file or directory). 2017/01/26 08:20:56.340421 [10931]: 60.nfs: Please try, as root, 'mount -t nfsd nfsd /proc/fs/nfsd' and then restart rpc.nfsd to correct the problem 2017/01/26 08:20:56.355286 [10931]: 60.nfs: [FAILED] 2017/01/26 08:20:56.375991 [10931]: startup event OK - enabling monitoring 2017/01/26 08:20:56.376044 [10931]: Set runstate to RUNNING (5) NFS_HOSTNAME is not configured. statd-callout failed 2017/01/26 08:20:58.654368 [10931]: monitor event OK - node re-enabled 2017/01/26 08:20:58.654421 [10931]: Node became HEALTHY. Ask recovery master to reallocate IPs In above output, I have blacklisted nfsd & sunrpc, but still somehow rpc.mountd is getting started which is interfering with gluster. After every ctdb restart gluster export list got empty. [root@gluserver601 ~]# showmount -e Export list for gluserver601: [root@gluserver601 ~]# By manually kill rpc.mountd & restarting glusterd process solve this issue. For long term ctdb startup need to be changed but for short term work around, we need to stop rpc.mountd getting started.
>2) There is also some amount of work needed to be done on >> the gNFSd process itself so that it can be brought up >> and down independently, IIRC from what Raghavendra Talur >> explained to me. This might be a little more work. >> --> Talur, maybe you can elaborate? > >I have no idea about the sizing of (2). Talur can you elaborate or delegate to >someone who knows? (->needinfo) IIRC, this requires a gluster volume command that can set nfs on/off for a volume. Sizing, this needs changes in Glusterd mainly and little more in CLI. I had explained a little to Jiffin and can continue helping him. It is not a very big change but is very intricate.
*** Bug 1436141 has been marked as a duplicate of this bug. ***