Bug 1154763 - cannot do anything in /tmp/ after running `rhc app tidy`
Summary: cannot do anything in /tmp/ after running `rhc app tidy`
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Containers
Version: 2.2.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Brenton Leanhardt
QA Contact: libra bugs
URL:
Whiteboard:
Depends On: 1146750
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-10-20 17:03 UTC by Brenton Leanhardt
Modified: 2014-12-10 13:24 UTC (History)
12 users (show)

Fixed In Version: pam_openshift-1.13.1.1-1
Doc Type: Bug Fix
Doc Text:
Previously, the polyinstanciated /tmp directory could be removed by running the "rhc app tidy" command, causing the /tmp directory to no longer function properly. This bug fix updates the pam_openshift module to prevent this command from removing gear /tmp directories. The module also now ensures that permissions are correctly set on gear /tmp directories. After applying this update, the ruby193-mcollective service must be restarted.
Clone Of: 1146750
Environment:
Last Closed: 2014-12-10 13:24:33 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2014:1979 0 normal SHIPPED_LIVE Red Hat OpenShift Enterprise 2.2.2 bug fix and enhancement update 2014-12-10 18:23:46 UTC

Description Brenton Leanhardt 2014-10-20 17:03:47 UTC
+++ This bug was initially created as a clone of Bug #1146750 +++

Description of problem:

After running `rhc app tidy`, the ability to create files or directories
in /tmp/ is lost.  A "No such file or directory" error occurs, even though
/tmp/ does exist.


Version-Release number of selected component (if applicable):


How reproducible:

Occurs with diy cartridge and the Haskell community cartridges, which were
the only cartridges tested.


Steps to Reproduce:
1. Create an app.
2. ssh in and confirm that files and directories can be created under /tmp/
3. Run `rhc app tidy` for the app.
4. ssh in and confirm that files and directories cannot be created.

Actual results:

[diy-ftweedal.rhcloud.com 5424c3d14382ec6c8e0000b3]\> touch /tmp/foo
touch: cannot touch `/tmp/foo': No such file or directory
[diy-ftweedal.rhcloud.com 5424c3d14382ec6c8e0000b3]\> ls /tmp/
[diy-ftweedal.rhcloud.com 5424c3d14382ec6c8e0000b3]\> ls -ld /tmp/
drwxrwxrwt. 0 5424c3d14382ec6c8e0000b3 root 6 Sep 25 21:40 /tmp/


Expected results:

Files and directories can still be created under /tmp/ after running
`rhc app tidy`.


Additional info:

--- Additional comment from Lili Nader on 2014-09-27 23:51:15 EDT ---

Works for me.  This is what I tried

1. create app
rhc app create -a diy -t diy
2. tidy app
rhc app tidy -a diy
3. ssh into gear and created test directory and file
mkdir /tmp/test
vi /tmp/test/test.txt
4. exit and run app tidy again
5. The file I created was deleted but I could create another file and directory.

--- Additional comment from Qixuan Wang on 2014-09-28 01:23:50 EDT ---

Test on devenv_5196, diy and python cartridge, it works for me, too.

1. Create an app
# rhc app create py33
2. Create files and directories on the gear
# rhc ssh py33
[py33-domain.dev.rhcloud.com 5427d00dc4219bd2a9000031]\> mkdir /tmp/test
[py33-domain.dev.rhcloud.com 5427d00dc4219bd2a9000031]\> touch /tmp/test.txt
[py33-domain.dev.rhcloud.com 5427d00dc4219bd2a9000031]\> ls -la /tmp
total 12
drwxrwxrwt.  3 5427d00dc4219bd2a9000031 root                     4096 Sep 28 05:10 .
dr-xr-xr-x. 26 root                     root                     4096 Sep 28 03:25 ..
drwx------.  2 5427d00dc4219bd2a9000031 5427d00dc4219bd2a9000031 4096 Sep 28 05:09 test
-rw-------.  1 5427d00dc4219bd2a9000031 5427d00dc4219bd2a9000031    0 Sep 28 05:10 test.txt
srwx------.  1 5427d00dc4219bd2a9000031 5427d00dc4219bd2a9000031    0 Sep 28 05:08 wsgi.25127.0.1.sock
3. Tidy the app
# rhc app tidy py33
4. Check the files and directories on the gear, they were deleted
[py33-domain.dev.rhcloud.com 5427d00dc4219bd2a9000031]\> ls -la /tmp
total 8
drwxrwxrwt.  2 5427d00dc4219bd2a9000031 root                     4096 Sep 28 05:11 .
dr-xr-xr-x. 26 root                     root                     4096 Sep 28 03:25 ..
srwx------.  1 5427d00dc4219bd2a9000031 5427d00dc4219bd2a9000031    0 Sep 28 05:11 wsgi.27283.0.1.sock
5. Create files and directories on the gear again
[py33-domain.dev.rhcloud.com 5427d00dc4219bd2a9000031]\> touch /tmp/file
[py33-domain.dev.rhcloud.com 5427d00dc4219bd2a9000031]\> mkdir /tmp/dir
[py33-domain.dev.rhcloud.com 5427d00dc4219bd2a9000031]\> ls -la /tmp
total 12
drwxrwxrwt.  3 5427d00dc4219bd2a9000031 root                     4096 Sep 28 05:12 .
dr-xr-xr-x. 26 root                     root                     4096 Sep 28 03:25 ..
drwx------.  2 5427d00dc4219bd2a9000031 5427d00dc4219bd2a9000031 4096 Sep 28 05:12 dir
-rw-------.  1 5427d00dc4219bd2a9000031 5427d00dc4219bd2a9000031    0 Sep 28 05:12 file
srwx------.  1 5427d00dc4219bd2a9000031 5427d00dc4219bd2a9000031    0 Sep 28 05:11 wsgi.27283.0.1.sock

The result is what we expect.

--- Additional comment from Fraser Tweedale on 2014-09-30 02:22:28 EDT ---

This is definitely still occurring for me.  Transcript with python-3.3 cartridge::


[dhcp-40-8:~/doc/blog] (ruby-2.1.2) [ master ] ftweedal% rhc app create py33 python-3.3 --no-git
Application Options
-------------------
Domain:     ftweedal
Cartridges: python-3.3
Gear Size:  default
Scaling:    no

Creating application 'py33' ... done


Waiting for your DNS name to be available ... done

Your application 'py33' is now available.

  URL:        http://py33-ftweedal.rhcloud.com/
  SSH to:     542a4b0f500446f46e000480.com
  Git remote: ssh://542a4b0f500446f46e000480.com/~/git/py33.git/

Run 'rhc show-app py33' for more details about your app.
[dhcp-40-8:~/doc/blog] (ruby-2.1.2) [ master ] ftweedal% rhc ssh py33
Connecting to 542a4b0f500446f46e000480.com ...
The authenticity of host 'py33-ftweedal.rhcloud.com (54.221.166.63)' can't be established.
RSA key fingerprint is cf:ee:77:cb:0e:fc:02:d7:72:7e:ae:80:c0:90:88:a7.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'py33-ftweedal.rhcloud.com,54.221.166.63' (RSA) to the list of known hosts.

    *********************************************************************

    You are accessing a service that is for use only by authorized users.
    If you do not have authorization, discontinue use at once.
    Any use of the services is subject to the applicable terms of the
    agreement which can be found at:
    https://www.openshift.com/legal

    *********************************************************************

    Welcome to OpenShift shell

    This shell will assist you in managing OpenShift applications.

    !!! IMPORTANT !!! IMPORTANT !!! IMPORTANT !!!
    Shell access is quite powerful and it is possible for you to
    accidentally damage your application.  Proceed with care!
    If worse comes to worst, destroy your application with "rhc app delete"
    and recreate it
    !!! IMPORTANT !!! IMPORTANT !!! IMPORTANT !!!

    Type "help" for more info.


[py33-ftweedal.rhcloud.com 542a4b0f500446f46e000480]\> mkdir /tmp/test
[py33-ftweedal.rhcloud.com 542a4b0f500446f46e000480]\> touch /tmp/test.txt
[py33-ftweedal.rhcloud.com 542a4b0f500446f46e000480]\> ls -la /tmp
total 12
drwxrwxrwt.  3 542a4b0f500446f46e000480 root                     4096 Sep 30 02:19 .
dr-xr-xr-x. 26 root                     root                     4096 Sep 25 04:42 ..
drwx------.  2 542a4b0f500446f46e000480 542a4b0f500446f46e000480 4096 Sep 30 02:19 test
-rw-------.  1 542a4b0f500446f46e000480 542a4b0f500446f46e000480    0 Sep 30 02:19 test.txt
srwx------.  1 542a4b0f500446f46e000480 542a4b0f500446f46e000480    0 Sep 30 02:18 wsgi.493463.0.1.sock
[py33-ftweedal.rhcloud.com 542a4b0f500446f46e000480]\> exit
Shared connection to py33-ftweedal.rhcloud.com closed.
[dhcp-40-8:~/doc/blog] (ruby-2.1.2) [ master ] ftweedal% rhc app tidy py33
RESULT:
py33 cleaned up
[dhcp-40-8:~/doc/blog] (ruby-2.1.2) [ master ] ftweedal% rhc ssh py33
Connecting to 542a4b0f500446f46e000480.com ...

    *********************************************************************

    You are accessing a service that is for use only by authorized users.
    If you do not have authorization, discontinue use at once.
    Any use of the services is subject to the applicable terms of the
    agreement which can be found at:
    https://www.openshift.com/legal

    *********************************************************************

    Welcome to OpenShift shell

    This shell will assist you in managing OpenShift applications.

    !!! IMPORTANT !!! IMPORTANT !!! IMPORTANT !!!
    Shell access is quite powerful and it is possible for you to
    accidentally damage your application.  Proceed with care!
    If worse comes to worst, destroy your application with "rhc app delete"
    and recreate it
    !!! IMPORTANT !!! IMPORTANT !!! IMPORTANT !!!

    Type "help" for more info.


[py33-ftweedal.rhcloud.com 542a4b0f500446f46e000480]\> ls -la /tmp
total 0
[py33-ftweedal.rhcloud.com 542a4b0f500446f46e000480]\> touch /tmp/file
touch: cannot touch `/tmp/file': No such file or directory
[py33-ftweedal.rhcloud.com 542a4b0f500446f46e000480]\> mkdir /tmp/dir
mkdir: cannot create directory `/tmp/dir': No such file or directory
[py33-ftweedal.rhcloud.com 542a4b0f500446f46e000480]\> exit
Shared connection to py33-ftweedal.rhcloud.com closed.
[dhcp-40-8:~/doc/blog] (ruby-2.1.2) [ master ] ftweedal% 


---

Am I doing something wrong?

--- Additional comment from Fraser Tweedale on 2014-09-30 03:13:28 EDT ---

Furthermore, I can report that after `rhc app tidy`:

[py33v2-ftweedal.rhcloud.com 542a56c64382ec5e160001d0]\> ls -ld /tmp
drwxrwx---. 0 542a56c64382ec5e160001d0 root 0 Sep 30 03:10 /tmp

Prior to tidy it was:

[py33v2-ftweedal.rhcloud.com 542a56c64382ec5e160001d0]\> ls -ld /tmp
drwxrwx---. 2 542a56c64382ec5e160001d0 root 4096 Sep 30 03:08 /tmp

So afaict, something is definitely awry.

--- Additional comment from Meng Bo on 2014-10-13 02:56:29 EDT ---

@Fraser Tweedale 
You should reopen the bug if it still can be reproduced for you, or else no one will care about this.

--- Additional comment from Jhon Honce on 2014-10-14 13:01:18 EDT ---

Please try again, the permissions to your gear's /tmp have been corrected.

--- Additional comment from openshift-github-bot on 2014-10-15 22:46:53 EDT ---

Commits pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/ea91dfd908e7ac48cf433b9a68a77ef24156767c
Bug 1146750 - Ensure permissions on temporary directories

https://github.com/openshift/origin-server/commit/c804db94db505f3610e9fcd7c16f2ae2214b235e
Bug 1146750 - Ensure permissions on temporary directories

* fix code format

--- Additional comment from Qixuan Wang on 2014-10-20 03:03:56 EDT ---

Tested on devenv_5247:

@Fraser Tweedale: I can reproduce your problem if I execute "rhc app tidy" and create directory/file under /tmp without terminating ssh.

@Jhon Honce: After "rhc app tidy", I reconnect the app and create directory/file under /tmp, it works well. Why "reconnect" is necessary? 

Please refer to the following results:

[py27s-d.dev.rhcloud.com 5444dc2fa822d05a92000003]\> mkdir /tmp/testdir
[py27s-d.dev.rhcloud.com 5444dc2fa822d05a92000003]\> touch /tmp/testfile
[py27s-d.dev.rhcloud.com 5444dc2fa822d05a92000003]\> ls -la /tmp
total 12
drwx-----T.  3 5444dc2fa822d05a92000003 root                     4096 Oct 20 06:42 .
dr-xr-xr-x. 26 root                     root                     4096 Oct 20 02:22 ..
-rw-r--r--.  1 5444dc2fa822d05a92000003 5444dc2fa822d05a92000003    0 Oct 20 05:56 haproxy_reload.lock
drwx------.  2 5444dc2fa822d05a92000003 5444dc2fa822d05a92000003 4096 Oct 20 06:42 testdir
-rw-------.  1 5444dc2fa822d05a92000003 5444dc2fa822d05a92000003    0 Oct 20 06:42 testfile
srwx------.  1 5444dc2fa822d05a92000003 5444dc2fa822d05a92000003    0 Oct 20 05:56 wsgi.7790.0.1.sock
[py27s-d.dev.rhcloud.com 5444dc2fa822d05a92000003]\> ls -la /tmp
total 0
[py27s-d.dev.rhcloud.com 5444dc2fa822d05a92000003]\> ls -ld /tmp
drwx-----T. 0 5444dc2fa822d05a92000003 root 0 Oct 20 06:43 /tmp
[py27s-d.dev.rhcloud.com 5444dc2fa822d05a92000003]\> mkdir /tmp/testdir2
mkdir: cannot create directory `/tmp/testdir2': No such file or directory
[py27s-d.dev.rhcloud.com 5444dc2fa822d05a92000003]\> touch /tmp/testfile2
touch: cannot touch `/tmp/testfile2': No such file or directory

[py27s-d.dev.rhcloud.com 5444dc2fa822d05a92000003]\> exit
exit
Connection to py27s-d.dev.rhcloud.com closed.
[root@localhost test]# rhc ssh py27s
/usr/local/rvm/gems/ruby-2.0.0-p481/gems/highline-1.6.21/lib/highline/system_extensions.rb:230: warning: Insecure world writable dir /usr/local/rvm/gems/ruby-2.0.0-p481@global in PATH, mode 040777
Connecting to 5444dc2fa822d05a92000003.rhcloud.com ...

    *********************************************************************

    You are accessing a service that is for use only by authorized users.
    If you do not have authorization, discontinue use at once.
    Any use of the services is subject to the applicable terms of the
    agreement which can be found at:
    https://www.openshift.com/legal

    *********************************************************************

    Welcome to OpenShift shell

    This shell will assist you in managing OpenShift applications.

    !!! IMPORTANT !!! IMPORTANT !!! IMPORTANT !!!
    Shell access is quite powerful and it is possible for you to
    accidentally damage your application.  Proceed with care!
    If worse comes to worst, destroy your application with "rhc app delete"
    and recreate it
    !!! IMPORTANT !!! IMPORTANT !!! IMPORTANT !!!

    Type "help" for more info.


[py27s-d.dev.rhcloud.com 5444dc2fa822d05a92000003]\> mkdir /tmp/testdir2
[py27s-d.dev.rhcloud.com 5444dc2fa822d05a92000003]\> touch /tmp/testfile2

--- Additional comment from Jhon Honce on 2014-10-20 12:19:05 EDT ---

Fixed in https://github.com/openshift/origin-server/pull/5889

Node will no longer remove PI /tmp directory during tidy.

Comment 3 Anping Li 2014-11-25 04:29:35 UTC
Verfied and pass on puddle-2-2-2014-11-24

1.create directory and file under /tmp.

[php53-anlidom.ose22-manual.com.cn tmp]\> ls -1
dir
file
haproxy_reload.lock


2. tidy app
rhc app tidy php53

3. we can found no files exsit in /tmp directoy after tidy
[php53-anlidom.ose22-manual.com.cn tmp]\> ls -lah
total 8.0K
drwx-----T.  2 5473f939e5fed547e7000027 root 4.0K Nov 24 21:22 .
dr-xr-xr-x. 25 root                     root 4.0K Nov 24 19:27 ..

3. In the same ssh session, create directory and files under /tmp, these files can be created.
[php53-anlidom.ose22-manual.com.cn tmp]\> mkdir dir1
[php53-anlidom.ose22-manual.com.cn tmp]\> touch file


[php53-anlidom.ose22-manual.com.cn tmp]\> ls -1
dir1
file1

Comment 5 errata-xmlrpc 2014-12-10 13:24:33 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2014-1979.html


Note You need to log in before you can comment on or make changes to this bug.