Bug 869695

Summary: SAPInstance resource agent should apply configured resource limits for SAP processes
Product: Red Hat Enterprise Linux 6 Reporter: Julio Entrena Perez <jentrena>
Component: resource-agentsAssignee: Chris Feist <cfeist>
Status: CLOSED ERRATA QA Contact: Cluster QE <mspqa-list>
Severity: high Docs Contact:
Priority: high    
Version: 6.4CC: agk, cluster-maint, ddumas, fdanapfe, jentrena, lhh, mjuricek
Target Milestone: rcKeywords: EasyFix, OtherQA, Patch
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: resource-agents-3.9.2-20.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 869705 (view as bug list) Environment:
Last Closed: 2013-02-21 02:52:52 EST Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
Bug Depends On:    
Bug Blocks: 782183, 869705, 886216    
Attachments:
Description Flags
Example cluster.conf
none
Example /usr/sap/sapservices
none
First draft of a proposed patch that applies resource limits from /usr/sap/sapservices before starting sapstartsrv none

Description Julio Entrena Perez 2012-10-24 11:11:01 EDT
Created attachment 632838 [details]
Example cluster.conf

> Description of problem:
SAP instances started by SAPInstance cluster resource agent inherit limits on system resources (e.g. max # of open file descriptors) for root user.
SAP instances need higher limits on the maximum number of open files (ulimit -n), the maximum stack size (ulimit -s) and the maximum size of data segments (ulimit -d).
Those limits can not be applied by PAM due to the way that SAP processes are started by cluster, see "Additional info" below.

> Version-Release number of selected component (if applicable):
resource-agents-3.9.2-12.el6

> How reproducible:
Always.

> Steps to Reproduce:
1. Configure a clustered SAP instance (see attached cluster.conf example).
2. Start the clustered service that includes the SAP instance (clusvcadm -e).
3. Observe current resource limits of the instance processes:

# ps -ef | grep jepadm |grep -v grep
jepadm    6032     1  0 Oct23 ?        00:00:18 /usr/sap/JEP/ASCS00/exe/sapstartsrv pf=/sapmnt/JEP/profile/JEP_ASCS00_jep-ascs -D -u jepadm
jepadm    6307     1  0 Oct23 ?        00:00:00 sapstart pf=/sapmnt/JEP/profile/JEP_ASCS00_jep-ascs
jepadm    6322  6307  0 Oct23 ?        00:00:01 ms.sapJEP_ASCS00 pf=/usr/sap/JEP/SYS/profile/JEP_ASCS00_jep-ascs
jepadm    6323  6307  0 Oct23 ?        00:00:23 en.sapJEP_ASCS00 pf=/usr/sap/JEP/SYS/profile/JEP_ASCS00_jep-ascs
  
> Actual results:
# for i in 6032 6307 6322 6323 ; do echo "$i:"; cat /proc/$i/limits | egrep "open|data|stack"; done
6032:
Max data size             unlimited            unlimited            bytes     
Max stack size            10485760             unlimited            bytes     
Max open files            1024                 4096                 files     
6307:
Max data size             unlimited            unlimited            bytes     
Max stack size            10485760             unlimited            bytes     
Max open files            1024                 4096                 files     
6322:
Max data size             unlimited            unlimited            bytes     
Max stack size            10485760             unlimited            bytes     
Max open files            1024                 4096                 files     
6323:
Max data size             unlimited            unlimited            bytes     
Max stack size            10485760             unlimited            bytes     
Max open files            1024                 4096                 files

> Expected results:
Limits specified in /usr/sap/sapservices are taken into account by SAPInstance RA:

# for i in 6032 6307 6322 6323 ; do echo "$i:"; cat /proc/$i/limits | egrep "open|data|stack"; done
6032:
Max data size             unlimited            unlimited            bytes     
Max stack size            268435456            268435456            bytes     
Max open files            65536                65536                files     
6307:
Max data size             unlimited            unlimited            bytes     
Max stack size            268435456            268435456            bytes     
Max open files            65536                65536                files     
6322:
Max data size             unlimited            unlimited            bytes     
Max stack size            268435456            268435456            bytes     
Max open files            65536                65536                files     
6323:
Max data size             unlimited            unlimited            bytes     
Max stack size            268435456            268435456            bytes     
Max open files            65536                65536                files

> Additional info:

Currently the following sequence of events occurs when rgmanager starts a SAP Instance:

- rgmanager starts SAPInstance RA as root.
- SAPInstance starts sapstartsrv as root with the account username as a parameter ('-u'):
195     $SAPSTARTSRV pf=$SAPSTARTPROFILE -D -u $sidadm
                                                ^
- sapstartsrv process starts as root and drops privileges by calling setgid() and setuid() (and then clones into new processes).
- SAPInstance calls sapcontrol to start the instance:
270     output=`$SAPCONTROL -nr $InstanceNr -function Start`

- sapcontrol instructs sapstartsrv to start SAP instance as described in [1].

Due to this sequence of events:
- PAM limits configured in /etc/security/limits.conf as described in SAP note 1496410 are not applied to processes started by rgmanager/SAPInstance RA.
- limits configured in /usr/sap/sapservices as described in SAP note 1437105 are not applied either since instances are not started by /etc/init.d/sapinit but by rgmanager/SAPInstance RA instead.

SAPInstance RA should take limits configured in /usr/sap/sapservices into account.
If no limits are specified in /usr/sap/sapservices then safe default limits should be applied.

[1] http://help.sap.com/saphelp_nw73ehp1/helpdata/en/b3/903925c34a45e28a2861b59c3c5623/content.htm
Comment 1 Julio Entrena Perez 2012-10-24 11:12:02 EDT
Created attachment 632842 [details]
Example /usr/sap/sapservices
Comment 5 Julio Entrena Perez 2012-10-25 12:55:26 EDT
Created attachment 633461 [details]
First draft of a proposed patch that applies resource limits from /usr/sap/sapservices before starting sapstartsrv

I'm attaching a first draft of a proposed patch.

If SAPInstance RA needs to start sapstartsrv, it will first check for resource limits configured at /usr/sap/sapservices and, if found, it will "eval" those before starting sapstartsrv, and will log the applied values with info level.

That's enough since processes of SAP instances will be started by sapstartsrv (at sapcontrol request) and will inherit the resource limits from it.
Comment 28 errata-xmlrpc 2013-02-21 02:52:52 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-0288.html