Bug 1365968

Summary: undefined class/module ActiveRecord::ConnectionAdapters::PostgreSQL::OID::Integer error killing generic and priorityworkers
Product: Red Hat CloudForms Management Engine Reporter: Jared Deubel <jdeubel>
Component: ApplianceAssignee: Gregg Tanzillo <gtanzill>
Status: CLOSED INSUFFICIENT_DATA QA Contact: Dave Johnson <dajohnso>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 5.6.0CC: abellott, jdeubel, jhardy, jrafanie, nicholas.schwaderer, obarenbo
Target Milestone: GA   
Target Release: 5.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-08-11 19:44:20 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jared Deubel 2016-08-10 15:58:00 UTC
Description of problem:

Upgraded from 5.5 to 5.6 and am now getting the below error consistently in the evm.log. 

Following steps have been performed:

- systemctl stop evmserverd
- yum update
- vmdb
- rake db:migrate
- rake evm:automate:reset
- systemctl start evmserverd

Snippet of evm.log
=========================================================================
[----] E, [2016-08-09T10:42:37.008774 #11561:6a7994] ERROR -- : MIQ(MiqPriorityWorker::Runner) ID [1000000016152] PID [11561] GUID [449c667e-5e3f-11e6-b939-005056b83c74] An error has occurred during work processing: MIQ(MiqPriorityWorker::Runner) "undefined class/module ActiveRecord::ConnectionAdapters::PostgreSQL::OID::Integer" attempting to get next message
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:67:in `rescue in block in get_message_via_drb'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:59:in `block in get_message_via_drb'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:36:in `loop'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:36:in `get_message_via_drb'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:87:in `get_message'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:150:in `block in do_work'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:146:in `loop'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:146:in `do_work'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:334:in `block in do_work_loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:331:in `loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:331:in `do_work_loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:153:in `run'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:128:in `start'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:21:in `start_worker'
/var/www/miq/vmdb/app/models/miq_worker.rb:346:in `block in start'
/opt/rh/cfme-gemset/gems/nakayoshi_fork-0.0.3/lib/nakayoshi_fork.rb:24:in `fork'
/opt/rh/cfme-gemset/gems/nakayoshi_fork-0.0.3/lib/nakayoshi_fork.rb:24:in `fork'
/var/www/miq/vmdb/app/models/miq_worker.rb:344:in `start'
/var/www/miq/vmdb/app/models/miq_worker.rb:274:in `start_worker'
/var/www/miq/vmdb/app/models/miq_worker.rb:154:in `block in sync_workers'
/var/www/miq/vmdb/app/models/miq_worker.rb:154:in `times'
/var/www/miq/vmdb/app/models/miq_worker.rb:154:in `sync_workers'
/var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:52:in `block in sync_workers'
/var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:50:in `each'
/var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:50:in `sync_workers'
/var/www/miq/vmdb/app/models/miq_server.rb:173:in `start'
/var/www/miq/vmdb/app/models/miq_server.rb:265:in `start'
/var/www/miq/vmdb/lib/workers/evm_server.rb:65:in `start'
/var/www/miq/vmdb/lib/workers/evm_server.rb:92:in `start'
/var/www/miq/vmdb/lib/workers/bin/evm_server.rb:4:in `<main>' Worker exiting.
=========================================================================

Version-Release number of selected component (if applicable):
5.6


Additional info:

Comment 2 Gregg Tanzillo 2016-08-10 16:11:46 UTC
Jared, can we get a dump of the customer's DB? If we can't get the entire DB then maybe just the miq_queue table. I'm thinking there may be a specific message in the queue that is causing this error.

Comment 4 Nick Schwaderer 2016-08-25 13:39:29 UTC
I don't even know what this is, but created an account just to make this comment!

This is the only place I have found that is getting - word for word - the exact same error as me. I'm working through a comprehensive upgrade of a decent sized 3 year old 7000+ commit Rails application and am hitting this error in one instance with the `audited` gem calling `associated_audits` on an object. I hope that helps you a bit, I will be keeping a close eye on this thread as this develops!

Comment 5 Joe Rafaniello 2017-03-15 12:37:48 UTC
FYI, Nick, I believe Gregg's comment in comment 2 is really the only way to solve this.  It hasn't been reported.  In our case, I'm guessing there's something in our miq_queue row (which contains serialized yaml columns) that prevent us from updating that row with the reported error.  Perhaps, the miq_queue table contained yaml serialized columns from a prior ruby and/or rails version and now it barfs on it.

We have to get a recreation table export to understand it fully.  Have you been able to track down your issue in https://github.com/collectiveidea/audited/issues/289?  It seems yours is with yaml columns and contents of columns.

Comment 6 Joe Rafaniello 2017-03-15 14:52:11 UTC
This commit in rails [1] removed ActiveRecord::ConnectionAdapters::PostgreSQL::OID::Integer.

Once we upgraded to rails 5.0.0, this class no longer exists.

It is possible that an existing miq_queue row created with rails 4.2 could have one of these private classes serialized in the args column.  Note, it's unclear why we're able to find the row but can't call update_attributes [2] on this row with the error:

ArgumentError: undefined class/module ActiveRecord::ConnectionAdapters::PostgreSQL::OID::Integer

[1] https://github.com/rails/rails/commit/aafee233fb3b4211ee0bfb1fca776c159bd1067e 
[2] https://github.com/ManageIQ/manageiq/blob/darga-5/app/models/miq_queue_worker_base/runner.rb#L60

Comment 7 Red Hat Bugzilla 2023-09-14 03:29:26 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days