Bug 988554 - oo-stats throws exception
oo-stats throws exception
Status: CLOSED CURRENTRELEASE
Product: OpenShift Online
Classification: Red Hat
Component: Pod (Show other bugs)
2.x
x86_64 Linux
unspecified Severity medium
: ---
: ---
Assigned To: Rajat Chopra
libra bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-07-25 16:05 EDT by Kenny Woodson
Modified: 2015-05-14 20:19 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-08-29 08:48:42 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Kenny Woodson 2013-07-25 16:05:25 EDT
Description of problem:

Trying to run oo-stats and this currently does not work.

----
]$ oo-stats -w 10 -d -f yaml 
/usr/sbin/oo-stats:585:in `block (2 levels) in get_db_stats': undefined method `[]' for nil:NilClass (NoMethodError)
	from /usr/sbin/oo-stats:582:in `each'
	from /usr/sbin/oo-stats:582:in `block in get_db_stats'
	from /usr/sbin/oo-stats:625:in `block (2 levels) in _with_each_record'
	from /opt/rh/ruby193/root/usr/local/share/gems/gems/mongo-1.8.1/lib/mongo/cursor.rb:286:in `each'
	from /usr/sbin/oo-stats:624:in `block in _with_each_record'
	from /opt/rh/ruby193/root/usr/local/share/gems/gems/mongo-1.8.1/lib/mongo/collection.rb:276:in `find'
	from /usr/sbin/oo-stats:623:in `_with_each_record'
	from /usr/sbin/oo-stats:569:in `get_db_stats'
	from /usr/sbin/oo-stats:109:in `block in gather_statistics'
	from /usr/sbin/oo-stats:98:in `time_msecs'
	from /usr/sbin/oo-stats:108:in `gather_statistics'
	from /usr/sbin/oo-stats:899:in `<main>'
----

Version-Release number of selected component (if applicable):

Current

How reproducible:
Very

Steps to Reproduce:
1. Run oo-stats.
2.
3.

Actual results:

Exception is thrown.

Expected results:

Should return a yaml output of openshift statistics.

Additional info:
Comment 1 Kenny Woodson 2013-07-25 16:08:50 EDT
I believe it was mentioned that not all applications have a component_instances.  This would return nil if it does not exist and therefore is probably relevant to the discovered bug.
Comment 2 Luke Meyer 2013-07-26 15:15:16 EDT
The oo-stats code in master has been refactored so the line numbers won't match up. I doubt it will be fixed by the refactor though. I also don't think the code in stage is the same as what you're looking at... can you indicate what code is at line 585 for you?
Comment 3 Kenny Woodson 2013-07-29 09:20:12 EDT
From the production code:

        app['component_instances'].each do |comp|
          carts_for_group[gid = comp['group_instance_id']] << (cart = comp['cartridge_name'])
          # from "foo-bar-1.1" we want "foo-bar" for the short name
[line # 585]short_carts_for_group[gid] << cart.match(/^  ([-\w]+)  -  [\d.]+  $/x)[1]
        end


Looks like a regex match on cart was unsuccessful which then performed an array lookup on a nil.
Comment 4 Luke Meyer 2013-07-29 09:58:13 EDT
Sounds right. I'm wondering if the assumption there about the format of comp['cartridge_name'] is broken for some carts, especially downloaded ones. This was written when all cart definitions were under our control...
Comment 5 Abhishek Gupta 2013-07-29 16:38:55 EDT
Rajat: Can you please follow up on this bug?
Comment 6 Rajat Chopra 2013-08-01 15:25:50 EDT
None of the downloadable apps in prod have the issue. Pending search - entire mongo database for any component_instance that has an invalid cartridge_name.
Comment 7 openshift-github-bot 2013-08-02 11:03:50 EDT
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/0bb58e5efeeccab4a6dc46b3b028937f0a67bef6
<admin/stats> should fix oo-stats bug 988554

https://bugzilla.redhat.com/show_bug.cgi?id=988554
oo-stats throws exception - not clear what cart name was causing the
break, but this should fix it.
Comment 8 Rajat Chopra 2013-08-02 13:13:23 EDT
Update :

Found the offending app/cartridge.
App broken - 51d6a23650044640b70002a2 : [#<BSON::OrderedHash:0x87d9bcb4 {"cartridge_name"=>"python-2.7"}>, #<BSON::OrderedHash:0x87d9bc28 {"cartridge_name"=>"ric-rdkit-FIXME"}>, #<BSON::OrderedHash:0x87d9bb88 {"cartridge_name"=>"cron-1.4"}>, #<BSON::OrderedHash:0x87d9bae8 {"cartridge_name"=>"smarterclayton-redis-2.6"}>]


Will investigate how this (ric-rdkit-FIXME) happened. Possible that version was put as FIXME and the validation was never there for it. Maybe the app needs a cleanup anyway.
Comment 9 Rajat Chopra 2013-08-16 13:47:40 EDT
The app has been already deleted. No such new case should creep in because we have validations in place now.
Comment 10 Jianwei Hou 2013-08-18 22:50:03 EDT
Verified on devenv_3671

Accoring to above comments, the broken state is generated by an application, now that the app is deleted, this should not happen again. So mark as verified.

The oo-stat is not broken
[root@ip-10-164-15-172 ~]# oo-stats -w 10 -d -f yaml
---
:timings_msecs:
  :load_broker_environment: 11643
  :get_node_entries: 10660
  :get_district_entries: 7
  :get_db_stats: 16
  :summarize_districts: 0
  :summarize_profiles: 0
:node_entries:
- &40952480
  :node_profile: small
  :district_uuid: NONE
  :district_active: false
  :max_active_gears: 90
  :gears_started_count: 1
  :gears_idle_count: 0
  :gears_stopped_count: 0
  :gears_deploying_count: 0
  :gears_unknown_count: 0
  :gears_total_count: 1
  :gears_active_count: 1
  :gears_usage_pct: 1.1111111111111112
  :gears_active_usage_pct: 1.1111111111111112
  :gears_active_pct: 100.0
  :id: !binary |-
    aXAtMTAtMTY0LTE1LTE3Mg==
  :name: !binary |-
    aXAtMTAtMTY0LTE1LTE3Mg==
:node_entries_hash:
  !binary "aXAtMTAtMTY0LTE1LTE3Mg==": *40952480
:district_entries: []
:district_entries_hash: {}
:district_summaries:
- &40950360
  :name: (NONE)
  :uuid: NONE profile=small
  :profile: small
  :district_capacity: 0
  :dist_avail_capacity: 0
  :dist_avail_uids: 0
  :nodes:
  - *40952480
  :missing_nodes: []
  :nodes_count: 1
  :nodes_active: 0
  :gears_started_count: 1
  :nodes_inactive: 1
  :gears_idle_count: 0
  :gears_stopped_count: 0
  :gears_deploying_count: 0
  :gears_unknown_count: 0
  :gears_total_count: 1
  :gears_active_count: 1
  :available_active_gears: 89
  :avg_active_usage_pct: 1.1111111111111112
  :dist_usage_pct: 0.0
  :lowest_active_usage_pct: 1.1111111111111112
  :highest_active_usage_pct: 1.1111111111111112
  :effective_available_gears: 0
  :gears_active_pct: 100.0
:district_summaries_hash:
  NONE profile=small: *40950360
:profile_summaries:
- &40975960
  :nodes_count: 1
  :nodes_active: 0
  :nodes_inactive: 1
  :gears_active_count: 1
  :gears_idle_count: 0
  :gears_stopped_count: 0
  :gears_unknown_count: 0
  :gears_total_count: 1
  :available_active_gears: 89
  :effective_available_gears: 0
  :avg_active_usage_pct: 1.1111111111111112
  :district_capacity: 0
  :dist_avail_capacity: 0
  :dist_avail_uids: 0
  :avg_dist_usage_pct: 0.0
  :profile: small
  :districts:
  - *40950360
  :missing_nodes: []
  :district_count: 1
  :gears_active_pct: 100.0
  :lowest_active_usage_pct: 1.1111111111111112
  :highest_active_usage_pct: 1.1111111111111112
  :lowest_dist_usage_pct: 0.0
  :highest_dist_usage_pct: 0.0
  :total_gears_in_db_records: 1
  :total_apps: 1
  :cartridges: &40972800
    jbossas-7: 1
    haproxy-1.4: 1
  :cartridges_short: &40972660
    jbossas: 1
    haproxy: 1
  :gear_count_db_minus_node: 0
:profile_summaries_hash:
  small: *40975960
:count_all:
  :apps: 1
  :gears: 1
  :cartridges:
    jbossas-7: 1
    haproxy-1.4: 1
  :cartridges_short:
    jbossas: 1
    haproxy: 1
  :users_with_num_apps:
    0: 5
    1: 1
  :users_with_num_gears:
    0: 5
    1: 1
  :nodes: 1
  :districts: 0
  :profiles: 1
:db_count_for_profile:
  small:
    :apps: 1
    :gears: 1
    :cartridges: *40972800
    :cartridges_short: *40972660
:db_count_per_user:
- :login: user_with_multiple_gear_sizes@test.com
  :apps: 0
  :gears: 0
- :login: user_with_extra_storage@test.com
  :apps: 0
  :gears: 0
- :login: user_with_certificate_capabilities@test.com
  :apps: 0
  :gears: 0
- :login: jhou+2@redhat.com
  :apps: 0
  :gears: 0
- :login: wsun+t1@redhat.com
  :apps: 0
  :gears: 0
- :login: yadu
  :small_apps: 1
  :apps: 1
  :small_gears: 1
  :gears: 1

Note You need to log in before you can comment on or make changes to this bug.