Bug 998345 - Occasionally the files for my gear under <cartridge>/metadata have incorrect owner
Occasionally the files for my gear under <cartridge>/metadata have incorrect ...
Status: CLOSED CURRENTRELEASE
Product: OpenShift Online
Classification: Red Hat
Component: Image (Show other bugs)
2.x
Unspecified Unspecified
high Severity medium
: ---
: ---
Assigned To: Andy Goldstein
libra bugs
: Reopened, UpcomingRelease
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-08-19 01:26 EDT by Meng Bo
Modified: 2015-05-14 20:34 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-01-23 22:22:54 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
gear_creation_log (24.16 KB, text/plain)
2013-08-19 01:26 EDT, Meng Bo
no flags Details

  None (edit)
Description Meng Bo 2013-08-19 01:26:38 EDT
Created attachment 787899 [details]
gear_creation_log

Description of problem:
Maybe this caused by some race condition issue. Occasionally found the files under my gear has incorrect owner. 

# ll 607221645069481718841344/php/metadata 521191db5052c00703000003/php/metadata
521191db5052c00703000003/php/metadata:
total 12
-rw-r--r--. 1 root 521191db5052c00703000003  642 Aug 18 23:33 jenkins_shell_command
-rw-r--r--. 1 root 607221645069481718841344  269 Aug 18 21:40 managed_files.yml
-rw-r--r--. 1 root 607221645069481718841344 2689 Aug 18 21:40 manifest.yml

607221645069481718841344/php/metadata:
total 12
-rw-r--r--. 1 root                     607221645069481718841344  642 Aug 18 23:33 jenkins_shell_command
-rw-r--r--. 1 607221645069481718841344 607221645069481718841344  269 Aug 18 21:40 managed_files.yml
-rw-r--r--. 1 607221645069481718841344 607221645069481718841344 2689 Aug 18 21:40 manifest.yml


The correct one should like this:
# ll 521190785052c00703000002/php/metadata
total 12
-rw-r--r--. 1 root 521190785052c00703000002  641 Aug 18 23:27 jenkins_shell_command
-rw-r--r--. 1 root 521190785052c00703000002  269 Aug 18 21:40 managed_files.yml
-rw-r--r--. 1 root 521190785052c00703000002 2689 Aug 18 21:40 manifest.yml


Version-Release number of selected component (if applicable):
devenv_3672

How reproducible:
occasionally

Steps to Reproduce:
1. Create about 10 apps at the same time may cause this issue.
# for i in `seq 1 10`; do rhc app create app3$i php-5 --no-dns & done
2.
3.

Actual results:
managed_files.yml and manifest.yml has incorrect owner for one or two gears.

Expected results:
All the files should have correct owner and permission.

Additional info:
Platform log for this two gears are attached.
Comment 1 Jhon Honce 2013-08-19 14:22:56 EDT
I attempted to recreate with devenv_3673:

# for i in `seq 1 10`; do rhc app create app3$i php-5 --no-dns & done
# for i in `seq 1 20`; do rhc app create app4$i php-5 --no-dns & done
# for i in `seq 1 10`; do rhc app create app5$i php-5 --no-dns & done

All 40 metadata directories were populated and secured as expected.

Please reopen if you can reproduce the issue.
Comment 2 Meng Bo 2013-09-11 02:43:57 EDT
Reproduce this on current INT env (devenv_3769) with high server load.

Myapp ruby19 cannot be started/stopped due to:
[ruby19-bmeng1int.int.rhcloud.com run]\> gear restart
Permission denied - /var/lib/openshift/52300c7003ef648a6e000560/ruby/metadata/manifest.yml

My gear info:
ruby19-bmeng1int.int.rhcloud.com

 hostname
ex-std-node3.int.rhcloud.com

GEAR_UUID=52300c7003ef648a6e000560

Account: bmeng+1@redhat.com

[ruby19-bmeng1int.int.rhcloud.com metadata]\> ls -al
total 20
drwxr-xr-x.  2 root                     52300c7003ef648a6e000560 4096 Sep 11 02:24 .
drwxr-xr-x. 13 52300c7003ef648a6e000560 52300c7003ef648a6e000560 4096 Sep 11 02:24 ..
-rw-r--r--.  1 root                     52300c7003ef648a6e000560  789 Sep 11 02:24 jenkins_shell_command
-rw-r--r--.  1 root                     52300c716cec0e040f000001  268 Sep  6 15:54 managed_files.yml
-rw-r--r--.  1 root                     52300c716cec0e040f000001 2469 Sep  6 15:54 manifest.yml


CC ops team to help collect useful logs if needed.
Comment 3 Andy Goldstein 2013-11-05 12:37:33 EST
https://github.com/openshift/origin-server/pull/4091
Comment 4 openshift-github-bot 2013-11-05 15:00:38 EST
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/fba14eaa04e7c5607f2b243faf80549f5a9695a1
Fix race condition in secure_cartridge

Remove Dir.chdir and use absolute file paths instead because chdir is
not thread-safe.

Bug 998345
Bug 1025043
Comment 5 Meng Bo 2013-11-06 23:32:29 EST
Tested on devenv_3998, with 20 apps creating at the same time.

Did not meet the files owner issue.

Move bug to verified.

Note You need to log in before you can comment on or make changes to this bug.