Bug 875949

Summary: teardown_pam_fs_limits.sh may fail on nodes with NFS mounts
Product: OKD Reporter: Jhon Honce <jhonce>
Component: ContainersAssignee: Jhon Honce <jhonce>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: high Docs Contact:
Priority: medium    
Version: 2.xCC: jhou, jinzhang, kraman, wdecoste
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 872379 Environment:
Last Closed: 2012-12-19 19:27:42 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 872379    
Bug Blocks:    

Description Jhon Honce 2012-11-12 22:21:28 UTC
+++ This bug was initially created as a clone of Bug #872379 +++

Description of problem:
If a node has any NFS mounts, the script may loop infinitely looking for the correct dev for GEAR_BASE_DIR.

Version-Release number of selected component (if applicable):
2.0.19

How reproducible:
Depends on order of mounts provided by df command

Steps to Reproduce:
1. Add NFS mount to a node
2. Create a gear
3.
  
Actual results:
If NFS mount is reported after correct mount script will loop infinitely

Expected results:
Script should ignore all NFS mounts as GEAR_BASE_DIR on an NFS mount is not supported.

Additional info:

--- Additional comment from jhonce on 2012-11-01 17:01:01 MST ---

Created attachment 636871 [details]
Screen shot providing additional information

--- Additional comment from jhonce on 2012-11-01 17:01:26 MST ---

Created attachment 636872 [details]
Screen shot providing additional information

--- Additional comment from jhonce on 2012-11-01 17:01:48 MST ---

Created attachment 636873 [details]
Screen shot providing additional information

--- Additional comment from rmillner on 2012-11-09 17:18:24 MST ---

The part that was failing turned out to be dead code anyway (results unused).  Cleaned up the script.

Pull Request:
https://github.com/openshift/origin-server/pull/866

--- Additional comment from jhou on 2012-11-12 04:51:32 MST ---

To verify this bug, QE need to mount an NFS on node, which may not be easy on a devenv instance.

So we verified this bug on our on-premise environment using above fix

Steps:
1. mount an NFS to node
2. rum setup_pam_fs_limits.sh 

[root@node openshift]# /bin/sh -x /usr/libexec/openshift/lib/setup_pam_fs_limits.sh 5abd74eee31d4008bf78185c486b9e64 0 0 0
+ LIMITSVARS='core data fsize memlock nofile rss stack cpu nproc as maxlogins priority locks sigpending msgqueue nice rprio'
+ source /etc/openshift/node.conf
++ PUBLIC_HOSTNAME=node.example.com
++ PUBLIC_IP=10.66.79.5
++ BROKER_HOST=broker.example.com
++ CLOUD_DOMAIN=example.com
++ GEAR_BASE_DIR=/var/lib/openshift
++ GEAR_SKEL_DIR=/etc/openshift/skel
++ GEAR_SHELL=/usr/bin/oo-trap-user
++ GEAR_GECOS='OpenShift guest'
++ GEAR_MIN_UID=500
++ GEAR_MAX_UID=6500
++ OPENSHIFT_NODE_PLUGINS=openshift-origin-node/plugins/unix_user_observer
++ CARTRIDGE_BASE_PATH=/usr/libexec/openshift/cartridges
++ LAST_ACCESS_DIR=/var/lib/openshift/.last_access
++ APACHE_ACCESS_LOG=/var/log/httpd/access_log
++ PROXY_MIN_PORT_NUM=35531
++ PROXY_PORTS_PER_GEAR=5
++ CREATE_APP_SYMLINKS=1
++ OPENSHIFT_HTTP_CONF_DIR=/etc/httpd/conf.d/openshift
+ limits_order=84
+ limits_nproc=100
+ quota_files=1000
++ expr 1024 '*' 128
+ quota_blocks=131072
+ source /etc/openshift/resource_limits.conf
++ node_profile=small
++ quota_files=40000
++ quota_blocks=1048576
++ max_apps=100
++ max_active_apps=100
++ limits_nproc=250
++ cpu_shares=128
++ cpu_cfs_quota_us=30000
++ memory_limit_in_bytes=536870912
++ memory_memsw_limit_in_bytes=641728512
++ apache_bandwidth='all 500000'
++ apache_maxconnection='all 20'
++ apache_bandwidtherror=510
++ rotatelogs_interval=86400
++ rotatelogs_format=-%Y%m%d-%H%M%S-%Z
+ username=5abd74eee31d4008bf78185c486b9e64
+ quota_blocks_custom=0
+ quota_files_custom=0
+ nproc_custom=0
+ '[' -n 0 ']'
+ '[' 0 -gt 1048576 ']'
+ '[' -n 0 ']'
+ '[' 0 -gt 40000 ']'
+ '[' -n 0 ']'
+ '[' 0 -le 250 ']'
+ limits_nproc=0
+ set_pam_limits 5abd74eee31d4008bf78185c486b9e64
+ USERNAME=5abd74eee31d4008bf78185c486b9e64
+ LIMITSFILE=/etc/security/limits.d/84-5abd74eee31d4008bf78185c486b9e64.conf
+ '[' -z '' ']'
+ cat
+ for KEY in '$LIMITSVARS'
++ eval echo '$limits_core'
+++ echo
+ VALUE=
+ '[' -n '' ']'
+ for KEY in '$LIMITSVARS'
++ eval echo '$limits_data'
+++ echo
+ VALUE=
+ '[' -n '' ']'
+ for KEY in '$LIMITSVARS'
++ eval echo '$limits_fsize'
+++ echo
+ VALUE=
+ '[' -n '' ']'
+ for KEY in '$LIMITSVARS'
++ eval echo '$limits_memlock'
+++ echo
+ VALUE=
+ '[' -n '' ']'
+ for KEY in '$LIMITSVARS'
++ eval echo '$limits_nofile'
+++ echo
+ VALUE=
+ '[' -n '' ']'
+ for KEY in '$LIMITSVARS'
++ eval echo '$limits_rss'
+++ echo
+ VALUE=
+ '[' -n '' ']'
+ for KEY in '$LIMITSVARS'
++ eval echo '$limits_stack'
+++ echo
+ VALUE=
+ '[' -n '' ']'
+ for KEY in '$LIMITSVARS'
++ eval echo '$limits_cpu'
+++ echo
+ VALUE=
+ '[' -n '' ']'
+ for KEY in '$LIMITSVARS'
++ eval echo '$limits_nproc'
+++ echo 0
+ VALUE=0
+ '[' -n 0 ']'
+ '[' -z '' ']'
+ echo '5abd74eee31d4008bf78185c486b9e64 hard nproc 0'
+ for KEY in '$LIMITSVARS'
++ eval echo '$limits_as'
+++ echo
+ VALUE=
+ '[' -n '' ']'
+ for KEY in '$LIMITSVARS'
++ eval echo '$limits_maxlogins'
+++ echo
+ VALUE=
+ '[' -n '' ']'
+ for KEY in '$LIMITSVARS'
++ eval echo '$limits_priority'
+++ echo
+ VALUE=
+ '[' -n '' ']'
+ for KEY in '$LIMITSVARS'
++ eval echo '$limits_locks'
+++ echo
+ VALUE=
+ '[' -n '' ']'
+ for KEY in '$LIMITSVARS'
++ eval echo '$limits_sigpending'
+++ echo
+ VALUE=
+ '[' -n '' ']'
+ for KEY in '$LIMITSVARS'
++ eval echo '$limits_msgqueue'
+++ echo
+ VALUE=
+ '[' -n '' ']'
+ for KEY in '$LIMITSVARS'
++ eval echo '$limits_nice'
+++ echo
+ VALUE=
+ '[' -n '' ']'
+ for KEY in '$LIMITSVARS'
++ eval echo '$limits_rprio'
+++ echo
+ VALUE=
+ '[' -n '' ']'
+ set_fs_quotas 5abd74eee31d4008bf78185c486b9e64 1048576 40000
+ quotas_enabled /var/lib/openshift
++ get_mountpoint /var/lib/openshift
++ awk '{ print $6 }'
++ tail -1
++ df -P /var/lib/openshift
+ QUOTA_ROOT=/
+ '[' -z / ']'
+ quotaon -u -p /
+ '[' 1 -eq 0 ']'
+ return 0
++ get_mountpoint /var/lib/openshift
++ awk '{ print $6 }'
++ tail -1
++ df -P /var/lib/openshift
+ setquota 5abd74eee31d4008bf78185c486b9e64 0 1048576 0 40000 /
[root@node openshift]# echo $?
0
 
There is not errors during execution, so move this bug to verified.

Comment 1 Jhon Honce 2012-11-12 23:27:40 UTC
Reproduce:

1) yum install nfs
2) mkdir /share
3) add /share  *(rw,no_root_squash) to /etc/exports
3) service rpcbind start
4) service nfs start
5) exportfs -a
6) mount -t nfs localhost:/share /mnt

7) Create gear
8) Delete gear


Fixed in https://github.com/openshift/origin-server/pull/877

Comment 2 Jianwei Hou 2012-11-13 05:55:20 UTC
Verified on devenv_2458

1) yum install nfs-utils
2) mkdir /share
3) add /share  *(rw,no_root_squash) to /etc/exports
3) service rpcbind start
4) service nfs start
5) exportfs -a
6) mount -t nfs localhost:/share /mnt
7) Create gear
rhc app create php1 php-5.3, gear is created successfully
8) Delete gear
rhc app delete php1, gear is deleted successfully