+++ This bug was initially created as a clone of Bug #1111163 +++ Description of problem: In check_nodes_cartridges function, it is calling "cart_list" facter, but this facter does not exist in ose-2.1 now, so that means, this check do nothing. This check need to be rewritten Version-Release number of selected component (if applicable): 2.1.z/2014-06-18.2 How reproducible: Always Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: --- Additional comment from Luke Meyer on 2014-06-19 09:18:36 EDT --- Agreed, this check should use mcollective to compare the actual manifest lists from each node (probably no need to compare to the broker anymore since the broker has its own concept of cartridges; maybe just ensure that all the cartridges the broker has imported are actually present on all the nodes). This check has long been the only thing looking at cart_list, but I didn't realize that fact was actually missing in 2.1.
PR: https://github.com/openshift/origin-server/pull/5599
Commit pushed to master at https://github.com/openshift/origin-server https://github.com/openshift/origin-server/commit/091dedc8b9758e87eebdbc87d2c293314a1aa6e7 oo-accept-systems: Fix check_nodes_cartridges Fix the check_nodes_cartridges test in oo-accept-systems to get the list of cartridges the correct way rather than trying to use the "cart_list" fact, which has been deprecated for several releases. With commit 4e94c47a381f3cfd168e9add0383414202aa96e5, the cart_list fact is no longer updated, so the check was essentially a no-op (no nodes would respond with the fact, so the carts_for Hash would remain empty, which would cause the test to terminate silently). This commit fixes bug 1118908.
Bug #1111163 already exists in OpenShift Enterprise to track this. Moving this to ON_QA since it's already merged and available for QE to verify.
This is failing on devenv_4998 [root@ip-10-67-172-194 ~]# oo-broker oo-accept-systems check_nodes_cartridges -v INFO: checking that each public_hostname resolves to external IP INFO: PUBLIC_HOSTNAME ec2-54-90-201-155.compute-1.amazonaws.com for ip-10-67-172-194 resolves to 10.67.172.194 INFO: checking that each public_hostname is unique INFO: checking that public_ip has been set for all nodes INFO: PUBLIC_IP 54.90.201.155 for ip-10-67-172-194 INFO: checking that public_ip is unique for all nodes INFO: checking that all node hosts have cartridges installed INFO: cartridges for ip-10-67-172-194: 10gen-mms-agent-0.1, cron-1.4, diy-0.1, haproxy-1.4, jbossas-7, jbosseap-6, jbossews-1.0, jbossews-2.0, jenkins-1, jenkins-client-1, metrics-0.1, mock-0.1, mock-0.2, mock-0.3, mock-0.4, mock-plugin-0.1, mock-plugin-0.2, mongodb-2.4, mysql-5.1, mysql-5.5, nodejs-0.10, nodejs-0.6, perl-5.10, php-5.3, php-5.4, phpmyadmin-4, postgresql-8.4, postgresql-9.2, python-2.6, python-2.7, python-3.3, rockmongo-1.1, ruby-1.8, ruby-1.9, ruby-2.0, switchyard-0, zend-5.6, zend-6.1 INFO: checking that same cartridges are installed on all node hosts INFO: checking that broker's list of cartridges is not stale FAIL: Couldn't retrieve cartridge list from broker: Network is unreachable - connect(2) 1 ERRORS I think the problem is oo-accept-systems is executed in docker container, and it is unable to curl the localhost that way. [root@ip-10-67-172-194 ~]# oo-broker bash-4.1# curl http://localhost:8080/broker/rest/cartridges.json curl: (7) Failed to connect to ::1: Network is unreachable bash-4.1# netstat -lntup Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name https://github.com/Miciah/origin-server/blob/2ff03e477686c7145fefda3d5d0b085a0cb51666/broker-util/oo-accept-systems#L185
I only tested the fix on an OSE installation and forgot that the devenvs have oo-broker. However, it looks like the bug is in code that follows my changes, and we only hit the bug now because I fixed bugs that prevented us from hitting this new issue. Presumably the broker is reachable, just not via localhost:8080. I'll look into this. Thanks!
oo-accept-systems needs to be run with oo-broker-attach. Unfortunately, a packaging issue is preventing oo-broker-attach from being installed in devenvs, so QA for this fix for Online is blocked until that issue is resolved. I will update this bug report when the issue is resolved so testing can proceed.
I have fixed the packaging issue. Testing should work now.
Verified on devenv_5027, funtion "check_nodes_cartridges" works, thanks. [root@ip-10-99-173-99 ~]# oo-broker-attach bash-4.1# oo-accept-systems check_nodes_cartridges -v INFO: checking that each public_hostname resolves to external IP INFO: PUBLIC_HOSTNAME ec2-54-87-79-239.compute-1.amazonaws.com for ip-10-99-173-99 resolves to 10.99.173.99 INFO: checking that each public_hostname is unique INFO: checking that public_ip has been set for all nodes INFO: PUBLIC_IP 54.87.79.239 for ip-10-99-173-99 INFO: checking that public_ip is unique for all nodes INFO: checking that all node hosts have cartridges installed INFO: cartridges for ip-10-99-173-99: 10gen-mms-agent-0.1, cron-1.4, diy-0.1, haproxy-1.4, jbossas-7, jbosseap-6, jbossews-1.0, jbossews-2.0, jenkins-1, jenkins-client-1, metrics-0.1, mock-0.1, mock-0.2, mock-0.3, mock-0.4, mock-plugin-0.1, mock-plugin-0.2, mongodb-2.4, mysql-5.1, mysql-5.5, nodejs-0.10, nodejs-0.6, perl-5.10, php-5.3, php-5.4, phpmyadmin-4, postgresql-8.4, postgresql-9.2, python-2.6, python-2.7, python-3.3, rockmongo-1.1, ruby-1.8, ruby-1.9, ruby-2.0, switchyard-0, zend-5.6, zend-6.1 INFO: checking that same cartridges are installed on all node hosts INFO: checking that broker's list of cartridges is not stale INFO: API reports carts: 10gen-mms-agent-0.1, cron-1.4, diy-0.1, haproxy-1.4, jboss-dv-6.0.0, jboss-vertx-2.1.1, jbossas-7, jbosseap-6, jbossews-1.0, jbossews-2.0, jenkins-1, jenkins-client-1, mock-0.1, mock-0.2, mock-0.3, mock-0.4, mock-plugin-0.1, mock-plugin-0.2, mongodb-2.4, mysql-5.1, mysql-5.5, nodejs-0.10, perl-5.10, php-5.3, php-5.4, phpmyadmin-4, postgresql-8.4, postgresql-9.2, python-2.6, python-2.7, python-3.3, rockmongo-1.1, ruby-1.8, ruby-1.9, ruby-2.0, switchyard-0, zend-6.1 FAIL: The broker's list of cartridges does not match what is available on the node hosts. Broker is missing: metrics-0.1, nodejs-0.6, zend-5.6 Broker has extra: jboss-dv-6.0.0, jboss-vertx-2.1.1 You can instruct the broker to pull an updated list of cartridges from a node and clear the management console's cache using the following commands: # oo-admin-ctl-cartridge -c import-node --activate --obsolete # oo-admin-console-cache --clear 1 ERRORS
Since there is mismatch, so I use the following commands to activate obsolete carts (metrics-0.1, nodejs-0.6, zend-5.6): # oo-admin-ctl-cartridge -c import-node --activate --obsolete # oo-admin-console-cache --clear Then check_nodes_cartridges, the result is the same with Comment 9. Is it acceptable?
The metrics-0.1, nodejs-0.6, and zend-5.6 cartridges are obsolete, and the broker does not list obsolete cartridges, but the node does. The jboss-dv-6.0.0 and jboss-vertx-2.1.1 cartridges are downloaded cartridges, so the node does not have them installed. It would make sense for oo-accept-systems to ignore obsolete and downloaded cartridges. Bug 1124938 addresses the problem. Thanks!