Bug 1118908

Summary: Function check_nodes_cartridges in oo-accept-systems script file doesn't work as expected.
Product: OpenShift Online Reporter: Miciah Dashiel Butler Masters <mmasters>
Component: PodAssignee: Miciah Dashiel Butler Masters <mmasters>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 2.xCC: bleanhar, jhonce, jhou, jokerman, libra-bugs, libra-onpremise-devel, lmeyer, mmasters, mmccomas, qixuan.wang, tdawson, yanpzhan
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1111163
: 1123999 1124938 (view as bug list) Environment:
Last Closed: 2014-10-10 00:48:41 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
Bug Depends On: 1123999    
Bug Blocks: 1111163, 1124938    

Description Miciah Dashiel Butler Masters 2014-07-11 21:39:24 UTC
+++ This bug was initially created as a clone of Bug #1111163 +++

Description of problem:
In check_nodes_cartridges function, it is calling "cart_list" facter, but this facter does not exist in ose-2.1 now, so that means, this check do nothing. This check need to be rewritten

Version-Release number of selected component (if applicable):
2.1.z/2014-06-18.2

How reproducible:
Always

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

--- Additional comment from Luke Meyer on 2014-06-19 09:18:36 EDT ---

Agreed, this check should use mcollective to compare the actual manifest lists from each node (probably no need to compare to the broker anymore since the broker has its own concept of cartridges; maybe just ensure that all the cartridges the broker has imported are actually present on all the nodes).

This check has long been the only thing looking at cart_list, but I didn't realize that fact was actually missing in 2.1.

Comment 1 Miciah Dashiel Butler Masters 2014-07-11 21:45:21 UTC
PR: https://github.com/openshift/origin-server/pull/5599

Comment 3 openshift-github-bot 2014-07-14 14:12:11 UTC
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/091dedc8b9758e87eebdbc87d2c293314a1aa6e7
oo-accept-systems: Fix check_nodes_cartridges

Fix the check_nodes_cartridges test in oo-accept-systems to get the list
of cartridges the correct way rather than trying to use the "cart_list"
fact, which has been deprecated for several releases.

With commit 4e94c47a381f3cfd168e9add0383414202aa96e5, the cart_list fact
is no longer updated, so the check was essentially a no-op (no nodes
would respond with the fact, so the carts_for Hash would remain empty,
which would cause the test to terminate silently).

This commit fixes bug 1118908.

Comment 4 Brenton Leanhardt 2014-07-22 17:11:56 UTC
Bug #1111163 already exists in OpenShift Enterprise to track this.  Moving this to ON_QA since it's already merged and available for QE to verify.

Comment 5 Jianwei Hou 2014-07-23 06:19:39 UTC
This is failing on devenv_4998 

[root@ip-10-67-172-194 ~]# oo-broker oo-accept-systems check_nodes_cartridges -v
INFO: checking that each public_hostname resolves to external IP
INFO: PUBLIC_HOSTNAME ec2-54-90-201-155.compute-1.amazonaws.com for ip-10-67-172-194 resolves to 10.67.172.194
INFO: checking that each public_hostname is unique
INFO: checking that public_ip has been set for all nodes
INFO: PUBLIC_IP 54.90.201.155 for ip-10-67-172-194
INFO: checking that public_ip is unique for all nodes
INFO: checking that all node hosts have cartridges installed
INFO: cartridges for ip-10-67-172-194: 10gen-mms-agent-0.1, cron-1.4, diy-0.1, haproxy-1.4, jbossas-7, jbosseap-6, jbossews-1.0, jbossews-2.0, jenkins-1, jenkins-client-1, metrics-0.1, mock-0.1, mock-0.2, mock-0.3, mock-0.4, mock-plugin-0.1, mock-plugin-0.2, mongodb-2.4, mysql-5.1, mysql-5.5, nodejs-0.10, nodejs-0.6, perl-5.10, php-5.3, php-5.4, phpmyadmin-4, postgresql-8.4, postgresql-9.2, python-2.6, python-2.7, python-3.3, rockmongo-1.1, ruby-1.8, ruby-1.9, ruby-2.0, switchyard-0, zend-5.6, zend-6.1
INFO: checking that same cartridges are installed on all node hosts
INFO: checking that broker's list of cartridges is not stale
FAIL: Couldn't retrieve cartridge list from broker: Network is unreachable - connect(2)
1 ERRORS

I think the problem is oo-accept-systems is executed in docker container, and it is unable to curl the localhost that way.

[root@ip-10-67-172-194 ~]# oo-broker
bash-4.1# curl http://localhost:8080/broker/rest/cartridges.json
curl: (7) Failed to connect to ::1: Network is unreachable

bash-4.1# netstat -lntup
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name

https://github.com/Miciah/origin-server/blob/2ff03e477686c7145fefda3d5d0b085a0cb51666/broker-util/oo-accept-systems#L185

Comment 6 Miciah Dashiel Butler Masters 2014-07-23 14:25:33 UTC
I only tested the fix on an OSE installation and forgot that the devenvs have oo-broker.  However, it looks like the bug is in code that follows my changes, and we only hit the bug now because I fixed bugs that prevented us from hitting this new issue.

Presumably the broker is reachable, just not via localhost:8080.  I'll look into this.  Thanks!

Comment 7 Miciah Dashiel Butler Masters 2014-07-28 20:02:37 UTC
oo-accept-systems needs to be run with oo-broker-attach.  Unfortunately, a packaging issue is preventing oo-broker-attach from being installed in devenvs, so QA for this fix for Online is blocked until that issue is resolved.  I will update this bug report when the issue is resolved so testing can proceed.

Comment 8 Troy Dawson 2014-07-29 14:14:30 UTC
I have fixed the packaging issue.  Testing should work now.

Comment 9 Qixuan Wang 2014-07-30 06:39:45 UTC
Verified on devenv_5027, funtion "check_nodes_cartridges" works, thanks.

[root@ip-10-99-173-99 ~]# oo-broker-attach
bash-4.1# oo-accept-systems check_nodes_cartridges -v
INFO: checking that each public_hostname resolves to external IP
INFO: PUBLIC_HOSTNAME ec2-54-87-79-239.compute-1.amazonaws.com for ip-10-99-173-99 resolves to 10.99.173.99
INFO: checking that each public_hostname is unique
INFO: checking that public_ip has been set for all nodes
INFO: PUBLIC_IP 54.87.79.239 for ip-10-99-173-99
INFO: checking that public_ip is unique for all nodes
INFO: checking that all node hosts have cartridges installed
INFO: cartridges for ip-10-99-173-99: 10gen-mms-agent-0.1, cron-1.4, diy-0.1, haproxy-1.4, jbossas-7, jbosseap-6, jbossews-1.0, jbossews-2.0, jenkins-1, jenkins-client-1, metrics-0.1, mock-0.1, mock-0.2, mock-0.3, mock-0.4, mock-plugin-0.1, mock-plugin-0.2, mongodb-2.4, mysql-5.1, mysql-5.5, nodejs-0.10, nodejs-0.6, perl-5.10, php-5.3, php-5.4, phpmyadmin-4, postgresql-8.4, postgresql-9.2, python-2.6, python-2.7, python-3.3, rockmongo-1.1, ruby-1.8, ruby-1.9, ruby-2.0, switchyard-0, zend-5.6, zend-6.1
INFO: checking that same cartridges are installed on all node hosts
INFO: checking that broker's list of cartridges is not stale
INFO: API reports carts: 10gen-mms-agent-0.1, cron-1.4, diy-0.1, haproxy-1.4, jboss-dv-6.0.0, jboss-vertx-2.1.1, jbossas-7, jbosseap-6, jbossews-1.0, jbossews-2.0, jenkins-1, jenkins-client-1, mock-0.1, mock-0.2, mock-0.3, mock-0.4, mock-plugin-0.1, mock-plugin-0.2, mongodb-2.4, mysql-5.1, mysql-5.5, nodejs-0.10, perl-5.10, php-5.3, php-5.4, phpmyadmin-4, postgresql-8.4, postgresql-9.2, python-2.6, python-2.7, python-3.3, rockmongo-1.1, ruby-1.8, ruby-1.9, ruby-2.0, switchyard-0, zend-6.1
FAIL:       The broker's list of cartridges does not match what is available on
      the node hosts.

      Broker is missing: metrics-0.1, nodejs-0.6, zend-5.6
      Broker has extra: jboss-dv-6.0.0, jboss-vertx-2.1.1

      You can instruct the broker to pull an updated list of cartridges from a node
      and clear the management console's cache using the following commands:

        # oo-admin-ctl-cartridge -c import-node --activate --obsolete
        # oo-admin-console-cache --clear

1 ERRORS

Comment 10 Qixuan Wang 2014-07-30 09:32:09 UTC
Since there is mismatch, so I use the following commands to activate obsolete carts (metrics-0.1, nodejs-0.6, zend-5.6):
# oo-admin-ctl-cartridge -c import-node --activate --obsolete
# oo-admin-console-cache --clear
Then check_nodes_cartridges, the result is the same with Comment 9. Is it acceptable?

Comment 11 Miciah Dashiel Butler Masters 2014-07-30 18:22:30 UTC
The metrics-0.1, nodejs-0.6, and zend-5.6 cartridges are obsolete, and the broker does not list obsolete cartridges, but the node does.

The jboss-dv-6.0.0 and jboss-vertx-2.1.1 cartridges are downloaded cartridges, so the node does not have them installed.

It would make sense for oo-accept-systems to ignore obsolete and downloaded cartridges.  Bug 1124938 addresses the problem.

Thanks!