Bug 995599
| Summary: | [oo-accept-node] no manifest in the cart repo matches | ||
|---|---|---|---|
| Product: | OpenShift Online | Reporter: | Kenny Woodson <kwoodson> |
| Component: | Containers | Assignee: | Jhon Honce <jhonce> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | libra bugs <libra-bugs> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 1.x | CC: | bmeng, kwoodson, sten, twiest |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2013-08-29 12:50:20 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Kenny Woodson
2013-08-09 20:00:57 UTC
Could mcollectived been restarted while starting? Not sure how this happened, but a simple stop / start of mcollective and it fixed this issue. platform-trace.log shows copy operations failing: August 09 16:37:10 INFO oo_spawn buffer(10/) /bin/cp: cannot create directory `/var/lib/openshift/.cartridge_repository/redhat-python/0.0.3/versions/2.7/template/libs': No such file or directory /bin/cp: preserving times for `/var/lib/openshift/.cartridge_repository/redhat-python/0.0.3/versions/2.7/template': No such file or directory /bin/cp: preserving times for `/var/lib/openshift/.cartridge_repository/redhat-python/0.0.3/versions/2.7': No such file or directory /bin/cp: preserving times for `/var/lib/openshift/.cartridge_repository/redhat-python/0.0.3/versions': No such file or directory August 07 14:19:23 INFO oo_spawn buffer(10/) /bin/cp: cannot create directory `/var/lib/openshift/.cartridge_repository/redhat-phpmyadmin/0.0.3/env': File exists August 07 14:19:23 INFO oo_spawn buffer(10/) /bin/cp: cannot create regular file `/var/lib/openshift/.cartridge_repository/redhat-phpmyadmin/0.0.3/versions/shared/conf.d/php.conf': File exists August 07 14:19:23 INFO oo_spawn buffer(10/) /bin/cp: cannot create regular file `/var/lib/openshift/.cartridge_repository/redhat-phpmyadmin/0.0.3/versions/shared/conf.d/openshift.conf.erb': File exists August 07 14:19:23 INFO oo_spawn buffer(10/) /bin/cp: cannot create regular file `/var/lib/openshift/.cartridge_repository/redhat-phpmyadmin/0.0.3/versions/shared/conf.d/phpMyAdmin.conf': File exists Cartridge Repository became corrupted. Added lock to openshift mcollective agent to prevent two agents from rebuilding cartridge repository at the same time. Commit pushed to master at https://github.com/openshift/origin-server https://github.com/openshift/origin-server/commit/55c3e065a6a3a8aeea57068f68cc6d579a419a33 Bug 995599 - Add lock when building cartridge repository * When building cartridge repository protect from an additional openshift mcollective being started Not sure if the following method is correct or not. 1. watch the process with grep key word mco # watch --interval 0.2 "ps -ef |grep mco|grep -v grep |grep -v update_yaml|grep -v ruby193" 2. do a parallel cartridge install # cd /usr/libexec/openshift/cartridges/ # for i in `ls`; do oo-admin-cartridge -a install -s ./$i/ --mco & done 3. chekc if there are more than one mco process generated. There still a lot of cartridge install processes exist at the same time. root 21733 21376 2 05:41 pts/1 00:00:00 ruby /usr/sbin/mco rpc -q openshift cartridge_repository action=install path=/usr/libexec/openshift/cartridges/jbossew root 21734 21365 2 05:41 pts/1 00:00:00 ruby /usr/sbin/mco rpc -q openshift cartridge_repository action=install path=/usr/libexec/openshift/cartridges/jbossas root 21737 21374 2 05:41 pts/1 00:00:00 ruby /usr/sbin/mco rpc -q openshift cartridge_repository action=install path=/usr/libexec/openshift/cartridges/10gen-m root 21738 21373 2 05:41 pts/1 00:00:00 ruby /usr/sbin/mco rpc -q openshift cartridge_repository action=install path=/usr/libexec/openshift/cartridges/diy root 21746 21386 2 05:41 pts/1 00:00:00 ruby /usr/sbin/mco rpc -q openshift cartridge_repository action=install path=/usr/libexec/openshift/cartridges/mock The method used is not verifying the issue. The issues is multiple mcollectived running multiple openshift agents. Not multiple mco clients. The only method I've seen to reproduce the problem and it's not reliable is to restart mcollectived repeatedly. /jwh Checked again on devenv_3678, after about 100 times repeatedly restart the mcollective service, there is only one mcollectived running. Move bug to verified. |