Bug 1021716
| Summary: | app-create and app-destroy failing on "No such file or directory" | ||
|---|---|---|---|
| Product: | OpenShift Online | Reporter: | Matt Woodson <mwoodson> |
| Component: | Containers | Assignee: | Rob Millner <rmillner> |
| Status: | CLOSED WORKSFORME | QA Contact: | libra bugs <libra-bugs> |
| Severity: | low | Docs Contact: | |
| Priority: | medium | ||
| Version: | 2.x | CC: | mfisher, mwoodson, rmillner, xtian |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2013-11-14 22:10:03 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
It appears that the File.open call in ApacheDB's flush method is getting Errno::ENOENT for geardb even though its not getting it for any other database which uses the same code block. This feels like a system issue. Need to see the output of: ls -ldZ /var/lib/openshift/.httpd.d /var/lib/openshift/.httpd.d/* ls -ldZ /etc/httpd/conf.d/openshift ...and the contents of /var/log/messages and /var/log/secure from Oct 21. Also, is there a more recent example of this error in the mcollective logs? Thanks! Dropping severity. It should be raised back up if there's a determination this is happening more than just on Oct 12'th at 08:12 AM. Please re-open if still experiencing this issue. The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days |
Description of problem: oo-accept-node is failing an alerting us to a problem with certain user id (uid's) not being cleaned up properly. Upon further investigation, this error is shown in the logs: "No such file or directory - /etc/httpd/conf.d/openshift/geardb.json-" Version-Release number of selected component (if applicable): rhc-node-1.15.4-1.el6oso.x86_64 How reproducible: Unsure Actual results: Leftover gear data that is unwanted. Expected results: Gear to be created and then removed properly. Additional info: Some relevent logs: ======================================================================== I, [2013-10-21T08:12:03.396715 #4474] INFO -- : openshift.rb:92:in `cartridge_do_action' cartridge_do_action call / action: cartridge_do, agen t=openshift, data={:cartridge=>"openshift-origin-node", :action=>"app-create", :args=> {"--with-app-uuid"=>"52651a06e0b8cd0fdd000016", "--with-app-name"=>"dev", "--with-container-uuid"=>"52651a06e0b8cd0fdd000016", "--with-container-name"=>"dev", "--with-namespace"=>"loadmylorry", "--with-uid"=>6138, "--with-request-id"=>"1f9e5f64ab7e7a1c0bd6b69bb3af1d1b", "--with-secret-token"=> "oOia9NxqDstFfhNa-C79A3VcQdPTpKEdiegaZ4fSl8sPQS5dOfKIGyDEH9zsy3AxR2u5K2p9yAgCiB71OoRxoXfLcaHWaXOghuZ3Vh6u1X64DwEhnVy426pGUvYXZMNo", "--cart-name"=>"openshift-origin-node"}, :process_results=>true} I, [2013-10-21T08:12:03.459722 #4474] INFO -- : openshift.rb:93:in `cartridge_do_action' cartridge_do_action validation = openshift-origin-node app-create {"--with-app-uuid"=>"52651a06e0b8cd0fdd000016", "--with-app-name"=>"dev", "--with-container-uuid"=>"52651a06e0b8cd0fdd000016", "--with-container-name"=>"dev", "--with-namespace"=>"loadmylorry", "--with-uid"=>6138, "--with-request-id"=>"1f9e5f64ab7e7a1c0bd6b69bb3af1d1b", "--with-secret-token"=>"oOia9NxqDstFfhNa-C79A3VcQdPTpKEdiegaZ4fSl8sPQS5dOfKIGyDEH9zsy3AxR2u5K2p9yAgCiB71OoRxoXfLcaHWaXOghuZ3Vh6u1X64DwEhnVy426pGUvYXZMNo", "--cart-name"=>"openshift-origin-node"} I, [2013-10-21T08:12:03.460211 #4474] INFO -- : openshift.rb:132:in `execute_action' Executing action [app-create] using method oo_app_create with args [{"--with-app-uuid"=>"52651a06e0b8cd0fdd000016", "--with-app-name"=>"dev", "--with-container-uuid"=>"52651a06e0b8cd0fdd000016", "--with-container-name"=>"dev", "--with-namespace"=>"loadmylorry", "--with-uid"=>6138, "--with-request-id"=>"1f9e5f64ab7e7a1c0bd6b69bb3af1d1b", "--with-secret-token"=>"oOia9NxqDstFfhNa-C79A3VcQdPTpKEdiegaZ4fSl8sPQS5dOfKIGyDEH9zsy3AxR2u5K2p9yAgCiB71OoRxoXfLcaHWaXOghuZ3Vh6u1X64DwEhnVy426pGUvYXZMNo", "--cart-name"=>"openshift-origin-node"}] I, [2013-10-21T08:12:05.180295 #4474] INFO -- : openshift.rb:330:in `rescue in oo_app_create' No such file or directory - /etc/httpd/conf.d/openshift/geardb.json- I, [2013-10-21T08:12:05.180657 #4474] INFO -- : openshift.rb:331:in `rescue in oo_app_create' ["/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.15.4/lib/openshift-origin-common/utils/file_needs_sync.rb:36:in `initialize'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.15.4/lib/openshift-origin-common/utils/file_needs_sync.rb:36:in `open'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.15.4/lib/openshift-origin-common/utils/file_needs_sync.rb:36:in `open'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-frontend-apachedb-0.1.4/lib/openshift/runtime/frontend/http/plugins/apachedb.rb:215:in `flush'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-frontend-apachedb-0.1.4/lib/openshift/runtime/frontend/http/plugins/apachedb.rb:240:in `close'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-frontend-apachedb-0.1.4/lib/openshift/runtime/frontend/http/plugins/apachedb.rb:265:in `open'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-frontend-apachedb-0.1.4/lib/openshift/runtime/frontend/http/plugins/apachedb.rb:300:in `create'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.11/lib/openshift-origin-node/model/frontend_httpd.rb:540:in `block in call_plugins'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.11/lib/openshift-origin-node/model/frontend_httpd.rb:536:in `map'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.11/lib/openshift-origin-node/model/frontend_httpd.rb:536:in `call_plugins'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.11/lib/openshift-origin-node/model/frontend_httpd.rb:181:in `create'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.11/lib/openshift-origin-node/model/application_container_ext/setup.rb:124:in `initialize_homedir'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-container-selinux-0.2.3/lib/openshift/runtime/containerization/selinux_container.rb:77:in `create'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.11/lib/openshift-origin-node/model/application_container.rb:197:in `block (2 levels) in create'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.15.4/lib/openshift-origin-common/utils/file_needs_sync.rb:38:in `block in open'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.15.4/lib/openshift-origin-common/utils/file_needs_sync.rb:36:in `open'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.15.4/lib/openshift-origin-common/utils/file_needs_sync.rb:36:in `open'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.11/lib/openshift-origin-node/model/application_container.rb:184:in `block in create'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.15.4/lib/openshift-origin-common/utils/file_needs_sync.rb:38:in `block in open'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.15.4/lib/openshift-origin-common/utils/file_needs_sync.rb:36:in `open'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.15.4/lib/openshift-origin-common/utils/file_needs_sync.rb:36:in `open'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.11/lib/openshift-origin-node/model/application_container.rb:177:in `create'", "/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:317:in `oo_app_create'", "/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:137:in: I, [2013-10-21T08:12:05.180813 #4474] INFO -- : openshift.rb:148:in `execute_action' Finished executing action [app-create] (-1) I, [2013-10-21T08:12:05.302861 #4474] INFO -- : openshift.rb:113:in `cartridge_do_action' cartridge_do_action failed (-1) ------ No such file or directory - /etc/httpd/conf.d/openshift/geardb.json- ------) I, [2013-10-21T08:12:05.405136 #4474] INFO -- : openshift.rb:92:in `cartridge_do_action' cartridge_do_action call / action: cartridge_do, agent=openshift, data={:cartridge=>"openshift-origin-node", :action=>"app-destroy", :args=> {"--with-app-uuid"=>"52651a06e0b8cd0fdd000016", "--with-app-name"=>"dev", "--with-container-uuid"=>"52651a06e0b8cd0fdd000016", "--with-container-name"=>"dev", "--with-namespace"=>"loadmylorry", "--with-uid"=>6138, "--with-request-id"=>"1f9e5f64ab7e7a1c0bd6b69bb3af1d1b", "--cart-name"=>"openshift-origin-node"}, :process_results=>true} I, [2013-10-21T08:12:05.405628 #4474] INFO -- : openshift.rb:93:in `cartridge_do_action' cartridge_do_action validation = openshift-origin-node app-destroy {"--with-app-uuid"=>"52651a06e0b8cd0fdd000016", "--with-app-name"=>"dev", "--with-container-uuid"=>"52651a06e0b8cd0fdd000016", "--with-container-name"=>"dev", "--with-namespace"=>"loadmylorry", "--with-uid"=>6138, "--with-request-id"=>"1f9e5f64ab7e7a1c0bd6b69bb3af1d1b", "--cart-name"=>"openshift-origin-node"} I, [2013-10-21T08:12:05.406310 #4474] INFO -- : openshift.rb:132:in `execute_action' Executing action [app-destroy] using method oo_app_destroy with args [{"--with-app-uuid"=>"52651a06e0b8cd0fdd000016", "--with-app-name"=>"dev", "--with-container-uuid"=>"52651a06e0b8cd0fdd000016", "--with-container-name"=>"dev", "--with-namespace"=>"loadmylorry", "--with-uid"=>6138, "--with-request-id"=>"1f9e5f64ab7e7a1c0bd6b69bb3af1d1b", "--cart-name"=>"openshift-origin-node"}] I, [2013-10-21T08:12:06.396340 #4474] INFO -- : openshift.rb:346:in `rescue in oo_app_destroy' No such file or directory - /etc/httpd/conf.d/openshift/geardb.json- I, [2013-10-21T08:12:06.396647 #4474] INFO -- : openshift.rb:347:in `rescue in oo_app_destroy' ["/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.15.4/lib/openshift-origin-common/utils/file_needs_sync.rb:36:in `initialize'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.15.4/lib/openshift-origin-common/utils/file_needs_sync.rb:36:in `open'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.15.4/lib/openshift-origin-common/utils/file_needs_sync.rb:36:in `open'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-frontend-apachedb-0.1.4/lib/openshift/runtime/frontend/http/plugins/apachedb.rb:215:in `flush'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-frontend-apachedb-0.1.4/lib/openshift/runtime/frontend/http/plugins/apachedb.rb:240:in `close'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-frontend-apachedb-0.1.4/lib/openshift/runtime/frontend/http/plugins/apachedb.rb:265:in `open'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-frontend-apachedb-0.1.4/lib/openshift/runtime/frontend/http/plugins/apachedb.rb:310:in `destroy'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.11/lib/openshift-origin-node/model/frontend_httpd.rb:540:in `block in call_plugins'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.11/lib/openshift-origin-node/model/frontend_httpd.rb:536:in `map'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.11/lib/openshift-origin-node/model/frontend_httpd.rb:536:in `call_plugins'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.11/lib/openshift-origin-node/model/frontend_httpd.rb:193:in `destroy'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-container-selinux-0.2.3/lib/openshift/runtime/containerization/selinux_container.rb:115:in `destroy'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.11/lib/openshift-origin-node/model/application_container.rb:255:in `block in destroy'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.15.4/lib/openshift-origin-common/utils/file_needs_sync.rb:38:in `block in open'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.15.4/lib/openshift-origin-common/utils/file_needs_sync.rb:36:in `open'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.15.4/lib/openshift-origin-common/utils/file_needs_sync.rb:36:in `open'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.11/lib/openshift-origin-node/model/application_container.rb:240:in `destroy'", "/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:343:in `oo_app_destroy'", "/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:137:in `execute_action'", "/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:104:in `cartridge_do_action'", "/opt/rh/ruby193/root/usr/share/ruby/mcollective/rpc/agent.rb:86:in `handlemsg'", "/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:126:in `block (2 levels) in dispatch'", "/opt/rh/ruby193/root/usr/share/ruby/timeout.rb:69:in `timeout'", "/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:125:in `block in dispatch'"] I, [2013-10-21T08:12:06.396825 #4474] INFO -- : openshift.rb:148:in `execute_action' Finished executing action [app-destroy] (-1) I, [2013-10-21T08:12:06.438783 #4474] INFO -- : openshift.rb:113:in `cartridge_do_action' cartridge_do_action failed (-1) ------ No such file or directory - /etc/httpd/conf.d/openshift/geardb.json- ------) I, [2013-10-21T08:12:06.690294 #4474] INFO -- : openshift.rb:92:in `cartridge_do_action' cartridge_do_action call / action: cartridge_do, agent=openshift, data={:cartridge=>"openshift-origin-node", :action=>"app-destroy", :args=> {"--with-app-uuid"=>"52651a06e0b8cd0fdd000016", "--with-app-name"=>"dev", "--with-container-uuid"=>"52651a06e0b8cd0fdd000016", "--with-container-name"=>"dev", "--with-namespace"=>"loadmylorry", "--with-uid"=>6138, "--with-request-id"=>"1f9e5f64ab7e7a1c0bd6b69bb3af1d1b", "--cart-name"=>"openshift-origin-node"}, :process_results=>true} I, [2013-10-21T08:12:06.690537 #4474] INFO -- : openshift.rb:93:in `cartridge_do_action' cartridge_do_action validation = openshift-origin-node app-destroy {"--with-app-uuid"=>"52651a06e0b8cd0fdd000016", "--with-app-name"=>"dev", "--with-container-uuid"=>"52651a06e0b8cd0fdd000016", "--with-container-name"=>"dev", "--with-namespace"=>"loadmylorry", "--with-uid"=>6138, "--with-request-id"=>"1f9e5f64ab7e7a1c0bd6b69bb3af1d1b", "--cart-name"=>"openshift-origin-node"} I, [2013-10-21T08:12:06.690876 #4474] INFO -- : openshift.rb:132:in `execute_action' Executing action [app-destroy] using method oo_app_destroy with args [{"--with-app-uuid"=>"52651a06e0b8cd0fdd000016", "--with-app-name"=>"dev", "--with-container-uuid"=>"52651a06e0b8cd0fdd000016", "--with-container-name"=>"dev", "--with-namespace"=>"loadmylorry", "--with-uid"=>6138, "--with-request-id"=>"1f9e5f64ab7e7a1c0bd6b69bb3af1d1b", "--cart-name"=>"openshift-origin-node"}] I, [2013-10-21T08:12:07.792715 #4474] INFO -- : openshift.rb:346:in `rescue in oo_app_destroy' No such file or directory - /etc/httpd/conf.d/openshift/geardb.json- I, [2013-10-21T08:12:07.793050 #4474] INFO -- : openshift.rb:347:in `rescue in oo_app_destroy' ["/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.15.4/lib/openshift-origin-common/utils/file_needs_sync.rb:36:in `initialize'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.15.4/lib/openshift-origin-common/utils/file_needs_sync.rb:36:in `open'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.15.4/lib/openshift-origin-common/utils/file_needs_sync.rb:36:in `open'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-frontend-apachedb-0.1.4/lib/openshift/runtime/frontend/http/plugins/apachedb.rb:215:in `flush'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-frontend-apachedb-0.1.4/lib/openshift/runtime/frontend/http/plugins/apachedb.rb:240:in `close'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-frontend-apachedb-0.1.4/lib/openshift/runtime/frontend/http/plugins/apachedb.rb:265:in `open'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-frontend-apachedb-0.1.4/lib/openshift/runtime/frontend/http/plugins/apachedb.rb:310:in `destroy'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.11/lib/openshift-origin-node/model/frontend_httpd.rb:540:in `block in call_plugins'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.11/lib/openshift-origin-node/model/frontend_httpd.rb:536:in `map'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.11/lib/openshift-origin-node/model/frontend_httpd.rb:536:in `call_plugins'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.11/lib/openshift-origin-node/model/frontend_httpd.rb:193:in `destroy'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-container-selinux-0.2.3/lib/openshift/runtime/containerization/selinux_container.rb:115:in `destroy'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.11/lib/openshift-origin-node/model/application_container.rb:255:in `block in destroy'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.15.4/lib/openshift-origin-common/utils/file_needs_sync.rb:38:in `block in open'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.15.4/lib/openshift-origin-common/utils/file_needs_sync.rb:36:in `open'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.15.4/lib/openshift-origin-common/utils/file_needs_sync.rb:36:in `open'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.11/lib/openshift-origin-node/model/application_container.rb:240:in `destroy'", "/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:343:in `oo_app_destroy'", "/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:137:in `execute_action'", "/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:104:in `cartridge_do_action'", "/opt/rh/ruby193/root/usr/share/ruby/mcollective/rpc/agent.rb:86:in `handlemsg'", "/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:126:in `block (2 levels) in dispatch'", "/opt/rh/ruby193/root/usr/share/ruby/timeout.rb:69:in `timeout'", "/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:125:in `block in dispatch'"] I, [2013-10-21T08:12:07.793201 #4474] INFO -- : openshift.rb:148:in `execute_action' Finished executing action [app-destroy] (-1) I, [2013-10-21T08:12:07.829684 #4474] INFO -- : openshift.rb:113:in `cartridge_do_action' cartridge_do_action failed (-1) ------ No such file or directory - /etc/httpd/conf.d/openshift/geardb.json-