Bug 1021716 - app-create and app-destroy failing on "No such file or directory" [NEEDINFO]
app-create and app-destroy failing on "No such file or directory"
Status: CLOSED WORKSFORME
Product: OpenShift Online
Classification: Red Hat
Component: Containers (Show other bugs)
2.x
Unspecified Unspecified
medium Severity low
: ---
: ---
Assigned To: Rob Millner
libra bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-10-21 17:33 EDT by Matt Woodson
Modified: 2015-05-14 19:31 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-11-14 17:10:03 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
rmillner: needinfo? (mwoodson)


Attachments (Terms of Use)

  None (edit)
Description Matt Woodson 2013-10-21 17:33:12 EDT
Description of problem:

oo-accept-node is failing an alerting us to a problem with certain user id (uid's) not being cleaned up properly.  Upon further investigation, this error is shown in the logs:

"No such file or directory - /etc/httpd/conf.d/openshift/geardb.json-"


Version-Release number of selected component (if applicable):
rhc-node-1.15.4-1.el6oso.x86_64

How reproducible:

Unsure


Actual results:

Leftover gear data that is unwanted.

Expected results:

Gear to be created and then removed properly.

Additional info:

Some relevent logs:
========================================================================
I, [2013-10-21T08:12:03.396715 #4474]  INFO -- : openshift.rb:92:in `cartridge_do_action' cartridge_do_action call / action: cartridge_do, agen
t=openshift, data={:cartridge=>"openshift-origin-node",
 :action=>"app-create",
 :args=>
  {"--with-app-uuid"=>"52651a06e0b8cd0fdd000016",
   "--with-app-name"=>"dev",
   "--with-container-uuid"=>"52651a06e0b8cd0fdd000016",
   "--with-container-name"=>"dev",
   "--with-namespace"=>"loadmylorry",
   "--with-uid"=>6138,
   "--with-request-id"=>"1f9e5f64ab7e7a1c0bd6b69bb3af1d1b",
   "--with-secret-token"=>
    "oOia9NxqDstFfhNa-C79A3VcQdPTpKEdiegaZ4fSl8sPQS5dOfKIGyDEH9zsy3AxR2u5K2p9yAgCiB71OoRxoXfLcaHWaXOghuZ3Vh6u1X64DwEhnVy426pGUvYXZMNo",
   "--cart-name"=>"openshift-origin-node"},
 :process_results=>true}

I, [2013-10-21T08:12:03.459722 #4474]  INFO -- : openshift.rb:93:in `cartridge_do_action' cartridge_do_action validation = openshift-origin-node app-create {"--with-app-uuid"=>"52651a06e0b8cd0fdd000016", "--with-app-name"=>"dev", "--with-container-uuid"=>"52651a06e0b8cd0fdd000016", "--with-container-name"=>"dev", "--with-namespace"=>"loadmylorry", "--with-uid"=>6138, "--with-request-id"=>"1f9e5f64ab7e7a1c0bd6b69bb3af1d1b", "--with-secret-token"=>"oOia9NxqDstFfhNa-C79A3VcQdPTpKEdiegaZ4fSl8sPQS5dOfKIGyDEH9zsy3AxR2u5K2p9yAgCiB71OoRxoXfLcaHWaXOghuZ3Vh6u1X64DwEhnVy426pGUvYXZMNo", "--cart-name"=>"openshift-origin-node"}
I, [2013-10-21T08:12:03.460211 #4474]  INFO -- : openshift.rb:132:in `execute_action' Executing action [app-create] using method oo_app_create with args [{"--with-app-uuid"=>"52651a06e0b8cd0fdd000016", "--with-app-name"=>"dev", "--with-container-uuid"=>"52651a06e0b8cd0fdd000016", "--with-container-name"=>"dev", "--with-namespace"=>"loadmylorry", "--with-uid"=>6138, "--with-request-id"=>"1f9e5f64ab7e7a1c0bd6b69bb3af1d1b", "--with-secret-token"=>"oOia9NxqDstFfhNa-C79A3VcQdPTpKEdiegaZ4fSl8sPQS5dOfKIGyDEH9zsy3AxR2u5K2p9yAgCiB71OoRxoXfLcaHWaXOghuZ3Vh6u1X64DwEhnVy426pGUvYXZMNo", "--cart-name"=>"openshift-origin-node"}]
I, [2013-10-21T08:12:05.180295 #4474]  INFO -- : openshift.rb:330:in `rescue in oo_app_create' No such file or directory - /etc/httpd/conf.d/openshift/geardb.json-
I, [2013-10-21T08:12:05.180657 #4474]  INFO -- : openshift.rb:331:in `rescue in oo_app_create' ["/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.15.4/lib/openshift-origin-common/utils/file_needs_sync.rb:36:in `initialize'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.15.4/lib/openshift-origin-common/utils/file_needs_sync.rb:36:in `open'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.15.4/lib/openshift-origin-common/utils/file_needs_sync.rb:36:in `open'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-frontend-apachedb-0.1.4/lib/openshift/runtime/frontend/http/plugins/apachedb.rb:215:in `flush'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-frontend-apachedb-0.1.4/lib/openshift/runtime/frontend/http/plugins/apachedb.rb:240:in `close'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-frontend-apachedb-0.1.4/lib/openshift/runtime/frontend/http/plugins/apachedb.rb:265:in `open'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-frontend-apachedb-0.1.4/lib/openshift/runtime/frontend/http/plugins/apachedb.rb:300:in `create'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.11/lib/openshift-origin-node/model/frontend_httpd.rb:540:in `block in call_plugins'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.11/lib/openshift-origin-node/model/frontend_httpd.rb:536:in `map'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.11/lib/openshift-origin-node/model/frontend_httpd.rb:536:in `call_plugins'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.11/lib/openshift-origin-node/model/frontend_httpd.rb:181:in `create'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.11/lib/openshift-origin-node/model/application_container_ext/setup.rb:124:in `initialize_homedir'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-container-selinux-0.2.3/lib/openshift/runtime/containerization/selinux_container.rb:77:in `create'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.11/lib/openshift-origin-node/model/application_container.rb:197:in `block (2 levels) in create'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.15.4/lib/openshift-origin-common/utils/file_needs_sync.rb:38:in `block in open'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.15.4/lib/openshift-origin-common/utils/file_needs_sync.rb:36:in `open'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.15.4/lib/openshift-origin-common/utils/file_needs_sync.rb:36:in `open'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.11/lib/openshift-origin-node/model/application_container.rb:184:in `block in create'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.15.4/lib/openshift-origin-common/utils/file_needs_sync.rb:38:in `block in open'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.15.4/lib/openshift-origin-common/utils/file_needs_sync.rb:36:in `open'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.15.4/lib/openshift-origin-common/utils/file_needs_sync.rb:36:in `open'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.11/lib/openshift-origin-node/model/application_container.rb:177:in `create'", "/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:317:in `oo_app_create'", "/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:137:in:
I, [2013-10-21T08:12:05.180813 #4474]  INFO -- : openshift.rb:148:in `execute_action' Finished executing action [app-create] (-1)
I, [2013-10-21T08:12:05.302861 #4474]  INFO -- : openshift.rb:113:in `cartridge_do_action' cartridge_do_action failed (-1)
------
No such file or directory - /etc/httpd/conf.d/openshift/geardb.json-
------)
I, [2013-10-21T08:12:05.405136 #4474]  INFO -- : openshift.rb:92:in `cartridge_do_action' cartridge_do_action call / action: cartridge_do, agent=openshift, data={:cartridge=>"openshift-origin-node",
 :action=>"app-destroy",
 :args=>
  {"--with-app-uuid"=>"52651a06e0b8cd0fdd000016",
   "--with-app-name"=>"dev",
   "--with-container-uuid"=>"52651a06e0b8cd0fdd000016",
   "--with-container-name"=>"dev",
   "--with-namespace"=>"loadmylorry",
   "--with-uid"=>6138,
   "--with-request-id"=>"1f9e5f64ab7e7a1c0bd6b69bb3af1d1b",
   "--cart-name"=>"openshift-origin-node"},
 :process_results=>true}

I, [2013-10-21T08:12:05.405628 #4474]  INFO -- : openshift.rb:93:in `cartridge_do_action' cartridge_do_action validation = openshift-origin-node app-destroy {"--with-app-uuid"=>"52651a06e0b8cd0fdd000016", "--with-app-name"=>"dev", "--with-container-uuid"=>"52651a06e0b8cd0fdd000016", "--with-container-name"=>"dev", "--with-namespace"=>"loadmylorry", "--with-uid"=>6138, "--with-request-id"=>"1f9e5f64ab7e7a1c0bd6b69bb3af1d1b", "--cart-name"=>"openshift-origin-node"}
I, [2013-10-21T08:12:05.406310 #4474]  INFO -- : openshift.rb:132:in `execute_action' Executing action [app-destroy] using method oo_app_destroy with args [{"--with-app-uuid"=>"52651a06e0b8cd0fdd000016", "--with-app-name"=>"dev", "--with-container-uuid"=>"52651a06e0b8cd0fdd000016", "--with-container-name"=>"dev", "--with-namespace"=>"loadmylorry", "--with-uid"=>6138, "--with-request-id"=>"1f9e5f64ab7e7a1c0bd6b69bb3af1d1b", "--cart-name"=>"openshift-origin-node"}]
I, [2013-10-21T08:12:06.396340 #4474]  INFO -- : openshift.rb:346:in `rescue in oo_app_destroy' No such file or directory - /etc/httpd/conf.d/openshift/geardb.json-
I, [2013-10-21T08:12:06.396647 #4474]  INFO -- : openshift.rb:347:in `rescue in oo_app_destroy' ["/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.15.4/lib/openshift-origin-common/utils/file_needs_sync.rb:36:in `initialize'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.15.4/lib/openshift-origin-common/utils/file_needs_sync.rb:36:in `open'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.15.4/lib/openshift-origin-common/utils/file_needs_sync.rb:36:in `open'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-frontend-apachedb-0.1.4/lib/openshift/runtime/frontend/http/plugins/apachedb.rb:215:in `flush'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-frontend-apachedb-0.1.4/lib/openshift/runtime/frontend/http/plugins/apachedb.rb:240:in `close'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-frontend-apachedb-0.1.4/lib/openshift/runtime/frontend/http/plugins/apachedb.rb:265:in `open'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-frontend-apachedb-0.1.4/lib/openshift/runtime/frontend/http/plugins/apachedb.rb:310:in `destroy'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.11/lib/openshift-origin-node/model/frontend_httpd.rb:540:in `block in call_plugins'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.11/lib/openshift-origin-node/model/frontend_httpd.rb:536:in `map'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.11/lib/openshift-origin-node/model/frontend_httpd.rb:536:in `call_plugins'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.11/lib/openshift-origin-node/model/frontend_httpd.rb:193:in `destroy'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-container-selinux-0.2.3/lib/openshift/runtime/containerization/selinux_container.rb:115:in `destroy'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.11/lib/openshift-origin-node/model/application_container.rb:255:in `block in destroy'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.15.4/lib/openshift-origin-common/utils/file_needs_sync.rb:38:in `block in open'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.15.4/lib/openshift-origin-common/utils/file_needs_sync.rb:36:in `open'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.15.4/lib/openshift-origin-common/utils/file_needs_sync.rb:36:in `open'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.11/lib/openshift-origin-node/model/application_container.rb:240:in `destroy'", "/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:343:in `oo_app_destroy'", "/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:137:in `execute_action'", "/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:104:in `cartridge_do_action'", "/opt/rh/ruby193/root/usr/share/ruby/mcollective/rpc/agent.rb:86:in `handlemsg'", "/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:126:in `block (2 levels) in dispatch'", "/opt/rh/ruby193/root/usr/share/ruby/timeout.rb:69:in `timeout'", "/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:125:in `block in dispatch'"]
I, [2013-10-21T08:12:06.396825 #4474]  INFO -- : openshift.rb:148:in `execute_action' Finished executing action [app-destroy] (-1)
I, [2013-10-21T08:12:06.438783 #4474]  INFO -- : openshift.rb:113:in `cartridge_do_action' cartridge_do_action failed (-1)
------
No such file or directory - /etc/httpd/conf.d/openshift/geardb.json-
------)
I, [2013-10-21T08:12:06.690294 #4474]  INFO -- : openshift.rb:92:in `cartridge_do_action' cartridge_do_action call / action: cartridge_do, agent=openshift, data={:cartridge=>"openshift-origin-node",
 :action=>"app-destroy",
 :args=>
  {"--with-app-uuid"=>"52651a06e0b8cd0fdd000016",
   "--with-app-name"=>"dev",
   "--with-container-uuid"=>"52651a06e0b8cd0fdd000016",
   "--with-container-name"=>"dev",
   "--with-namespace"=>"loadmylorry",
   "--with-uid"=>6138,
   "--with-request-id"=>"1f9e5f64ab7e7a1c0bd6b69bb3af1d1b",
   "--cart-name"=>"openshift-origin-node"},
 :process_results=>true}

I, [2013-10-21T08:12:06.690537 #4474]  INFO -- : openshift.rb:93:in `cartridge_do_action' cartridge_do_action validation = openshift-origin-node app-destroy {"--with-app-uuid"=>"52651a06e0b8cd0fdd000016", "--with-app-name"=>"dev", "--with-container-uuid"=>"52651a06e0b8cd0fdd000016", "--with-container-name"=>"dev", "--with-namespace"=>"loadmylorry", "--with-uid"=>6138, "--with-request-id"=>"1f9e5f64ab7e7a1c0bd6b69bb3af1d1b", "--cart-name"=>"openshift-origin-node"}
I, [2013-10-21T08:12:06.690876 #4474]  INFO -- : openshift.rb:132:in `execute_action' Executing action [app-destroy] using method oo_app_destroy with args [{"--with-app-uuid"=>"52651a06e0b8cd0fdd000016", "--with-app-name"=>"dev", "--with-container-uuid"=>"52651a06e0b8cd0fdd000016", "--with-container-name"=>"dev", "--with-namespace"=>"loadmylorry", "--with-uid"=>6138, "--with-request-id"=>"1f9e5f64ab7e7a1c0bd6b69bb3af1d1b", "--cart-name"=>"openshift-origin-node"}]
I, [2013-10-21T08:12:07.792715 #4474]  INFO -- : openshift.rb:346:in `rescue in oo_app_destroy' No such file or directory - /etc/httpd/conf.d/openshift/geardb.json-
I, [2013-10-21T08:12:07.793050 #4474]  INFO -- : openshift.rb:347:in `rescue in oo_app_destroy' ["/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.15.4/lib/openshift-origin-common/utils/file_needs_sync.rb:36:in `initialize'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.15.4/lib/openshift-origin-common/utils/file_needs_sync.rb:36:in `open'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.15.4/lib/openshift-origin-common/utils/file_needs_sync.rb:36:in `open'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-frontend-apachedb-0.1.4/lib/openshift/runtime/frontend/http/plugins/apachedb.rb:215:in `flush'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-frontend-apachedb-0.1.4/lib/openshift/runtime/frontend/http/plugins/apachedb.rb:240:in `close'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-frontend-apachedb-0.1.4/lib/openshift/runtime/frontend/http/plugins/apachedb.rb:265:in `open'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-frontend-apachedb-0.1.4/lib/openshift/runtime/frontend/http/plugins/apachedb.rb:310:in `destroy'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.11/lib/openshift-origin-node/model/frontend_httpd.rb:540:in `block in call_plugins'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.11/lib/openshift-origin-node/model/frontend_httpd.rb:536:in `map'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.11/lib/openshift-origin-node/model/frontend_httpd.rb:536:in `call_plugins'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.11/lib/openshift-origin-node/model/frontend_httpd.rb:193:in `destroy'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-container-selinux-0.2.3/lib/openshift/runtime/containerization/selinux_container.rb:115:in `destroy'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.11/lib/openshift-origin-node/model/application_container.rb:255:in `block in destroy'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.15.4/lib/openshift-origin-common/utils/file_needs_sync.rb:38:in `block in open'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.15.4/lib/openshift-origin-common/utils/file_needs_sync.rb:36:in `open'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.15.4/lib/openshift-origin-common/utils/file_needs_sync.rb:36:in `open'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.11/lib/openshift-origin-node/model/application_container.rb:240:in `destroy'", "/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:343:in `oo_app_destroy'", "/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:137:in `execute_action'", "/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:104:in `cartridge_do_action'", "/opt/rh/ruby193/root/usr/share/ruby/mcollective/rpc/agent.rb:86:in `handlemsg'", "/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:126:in `block (2 levels) in dispatch'", "/opt/rh/ruby193/root/usr/share/ruby/timeout.rb:69:in `timeout'", "/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:125:in `block in dispatch'"]
I, [2013-10-21T08:12:07.793201 #4474]  INFO -- : openshift.rb:148:in `execute_action' Finished executing action [app-destroy] (-1)
I, [2013-10-21T08:12:07.829684 #4474]  INFO -- : openshift.rb:113:in `cartridge_do_action' cartridge_do_action failed (-1)
------
No such file or directory - /etc/httpd/conf.d/openshift/geardb.json-
Comment 1 Rob Millner 2013-10-29 18:50:50 EDT
It appears that the File.open call in ApacheDB's flush method is getting Errno::ENOENT for geardb even though its not getting it for any other database which uses the same code block.

This feels like a system issue.

Need to see the output of:
ls -ldZ /var/lib/openshift/.httpd.d /var/lib/openshift/.httpd.d/*

ls -ldZ /etc/httpd/conf.d/openshift

...and the contents of /var/log/messages and /var/log/secure from Oct 21.  Also, is there a more recent example of this error in the mcollective logs?

Thanks!
Comment 2 Rob Millner 2013-10-29 18:54:27 EDT
Dropping severity.  It should be raised back up if there's a determination this is happening more than just on Oct 12'th at 08:12 AM.
Comment 3 Rob Millner 2013-11-14 17:10:03 EST
Please re-open if still experiencing this issue.

Note You need to log in before you can comment on or make changes to this bug.