Bug 1246667

Summary: katello-reindex fails on errata can not handle bad errata
Product: Red Hat Satellite Reporter: Kathryn Dixon <kdixon>
Component: Errata ManagementAssignee: Partha Aji <paji>
Status: CLOSED ERRATA QA Contact: Sachin Ghai <sghai>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.1.0CC: bbuckingham, cwelton, kdixon, mmccune, parmstro, sghai, tkolhar
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
URL: http://projects.theforeman.org/issues/11140
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-09-15 07:22:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
cancelled sync task none

Description Kathryn Dixon 2015-07-24 20:35:16 UTC
Description of problem:

After killing a synchronization task, the foreman-rake katello:reindex fails at errata, I had customer run the foreman-rake katello:clean_backend_objects --trace which does not fail. then redo the reindex and fails at same error.

Version-Release number of selected component (if applicable):


How reproducible: 50%


Steps to Reproduce:
1. sat 6.1 beta
2. sync a task
3. kill task while syncing

Actual results:

oreman-rake katello:reindex --trace
** Invoke katello:reindex (first_time)
** Invoke environment (first_time)
** Execute environment
** Invoke katello:check_ping (first_time)
** Execute katello:check_ping
** Invoke katello:reset_backends:elasticsearch (first_time)
** Invoke environment
** Execute katello:reset_backends:elasticsearch
Elasticsearch Indices cleared.
** Execute katello:reindex
Re-indexing Katello::ActivationKey
Re-indexing Katello::ContentView
Re-indexing Katello::ContentViewFilter
Re-indexing Katello::ContentViewErratumFilterRule
Re-indexing Katello::TaskStatus
Re-indexing Katello::ContentViewHistory
Re-indexing Katello::Product
Re-indexing Katello::Repository
Re-indexing Katello::Provider
Re-indexing Katello::ContentViewPackageFilterRule
Re-indexing Katello::ContentViewPackageGroupFilterRule
Re-indexing Katello::ContentViewPuppetEnvironment
Re-indexing Katello::ContentViewPuppetModule
Re-indexing Katello::System
Re-indexing Katello::Distributor
Re-indexing Katello::HostCollection
Re-indexing Katello::Job
Re-indexing Katello::Notice
Re-indexing Katello::ContentViewErratumFilter
Re-indexing Katello::ContentViewPackageFilter
Re-indexing Katello::ContentViewPackageGroupFilter
Re-indexing Katello::PulpTaskStatus
Re-indexing Katello::PulpSyncStatus
Re-indexing Katello::MarketingProduct
Re-indexing Katello::Hypervisor
Re-indexing Katello::Package
Re-indexing Katello::PuppetModule
Re-indexing Katello::Distribution
Re-indexing Katello::PackageGroup
Re-indexing Katello::Erratum
rake aborted!
undefined method `each' for #<Class:0x0000000a465ff0>
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/dynamic_matchers.rb:55:in `method_missing'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.51/lib/katello/tasks/reindex.rake:40:in `rescue in index_objects'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.51/lib/katello/tasks/reindex.rake:36:in `index_objects'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.51/lib/katello/tasks/reindex.rake:132:in `block (2 levels) in <top (required)>'
/opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:205:in `call'
/opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:205:in `block in execute'
/opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:200:in `each'
/opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:200:in `execute'
/opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:158:in `block in invoke_with_call_chain'
/opt/rh/ruby193/root/usr/share/ruby/monitor.rb:211:in `mon_synchronize'
/opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:151:in `invoke_with_call_chain'
/opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:144:in `invoke'
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:116:in `invoke_task'
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:in `block (2 levels) in top_level'
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:in `each'
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:in `block in top_level'
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:133:in `standard_exception_handling'
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:88:in `top_level'
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:66:in `block in run'
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:133:in `standard_exception_handling'
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:63:in `run'
/opt/rh/ruby193/root/usr/bin/rake:32:in `<main>'
caused by: (NoMethodError) undefined method `id' for nil:NilClass
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.51/app/models/katello/glue/pulp/pulp_content_unit.rb:80:in `block in sync_repository_associations'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.51/app/models/katello/glue/pulp/pulp_content_unit.rb:80:in `map'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.51/app/models/katello/glue/pulp/pulp_content_unit.rb:80:in `sync_repository_associations'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.51/app/models/katello/glue/pulp/pulp_content_unit.rb:113:in `block (2 levels) in update_repository_associations'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.51/app/models/katello/glue/pulp/pulp_content_unit.rb:112:in `each'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.51/app/models/katello/glue/pulp/pulp_content_unit.rb:112:in `block in update_repository_associations'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/connection_adapters/abstract/database_statements.rb:192:in `transaction'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:208:in `transaction'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.51/app/models/katello/glue/pulp/pulp_content_unit.rb:103:in `update_repository_associations'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.51/app/models/katello/glue/pulp/pulp_content_unit.rb:60:in `import_all'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.51/lib/katello/tasks/reindex.rake:133:in `block (3 levels) in <top (required)>'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.51/lib/katello/tasks/reindex.rake:37:in `index_objects'
    ... skipped 19 lines
Tasks: TOP => katello:reindex


Expected results:

Successful reindex

Additional info:

upstream bug found

http://projects.theforeman.org/issues/11140

The reindex task assumes that [Katello::Package, Katello::PuppetModule, Katello::Distribution, Katello::PackageGroup, Katello::Erratum] all have an "each" method in the case of on error. This causes an additional error thrown every time indexing one of those items fail. We just need to report the exception at the bulk level in those cases and not try to index one at a time.

Also we need to handle errata that point to repo memberships that do not exist in katello (as in we need to ignore those associations). This can happen either when user directly create/syncs a private repo via pulp-admin OR deletes a repo and for some reason the membership did not get updated.

Comment 3 Tazim Kolhar 2015-09-03 06:39:02 UTC
Hi,

  I have a query here for last step:
   3. kill task while syncing

  it is not possible to kill the task in UI . we can stop the 
  task but not kill .

   please,guide here
   thanks in advance
Thanks and Regards,
Tazim

Comment 4 Sachin Ghai 2015-09-09 09:24:29 UTC
Verified with sat6.1.2 snap1

- Installed new snap1 (sat6.1.2).
- Configured external capsule
- Synced couple of RH repos.
- Synced rhel7 repo and cancel the sync task from Monitor-tasks
- ran foreman-rake katello:reindex --trace

"foreman-rake katello:reindex --trace" cmd completed successfully. I didn't get any exception.

[root@cloud-qe-22 ~]# foreman-rake katello:reindex --trace
** Invoke katello:reindex (first_time)
** Invoke environment (first_time)
** Execute environment
** Invoke katello:check_ping (first_time)
** Execute katello:check_ping
** Invoke katello:reset_backends:elasticsearch (first_time)
** Invoke environment 
** Execute katello:reset_backends:elasticsearch
Elasticsearch Indices cleared.
** Execute katello:reindex
Re-indexing Katello::ActivationKey
Re-indexing Katello::ContentView
Re-indexing Katello::ContentViewFilter
Re-indexing Katello::Product
Re-indexing Katello::Repository
Re-indexing Katello::ContentViewErratumFilterRule
Re-indexing Katello::TaskStatus
Re-indexing Katello::ContentViewHistory
Re-indexing Katello::Provider
Re-indexing Katello::ContentViewPackageFilterRule
Re-indexing Katello::ContentViewPackageGroupFilterRule
Re-indexing Katello::ContentViewPuppetEnvironment
Re-indexing Katello::System
Re-indexing Katello::ContentViewPuppetModule
Re-indexing Katello::Distributor
Re-indexing Katello::HostCollection
Re-indexing Katello::Job
Re-indexing Katello::Notice
Re-indexing Katello::ContentViewErratumFilter
Re-indexing Katello::ContentViewPackageFilter
Re-indexing Katello::ContentViewPackageGroupFilter
Re-indexing Katello::MarketingProduct
Re-indexing Katello::PulpTaskStatus
Re-indexing Katello::PulpSyncStatus
Re-indexing Katello::Hypervisor
Re-indexing Katello::Package
Re-indexing Katello::PuppetModule
Re-indexing Katello::Distribution
Re-indexing Katello::PackageGroup
Re-indexing Katello::Erratum
[root@cloud-qe-22 ~]#

Comment 5 Sachin Ghai 2015-09-09 09:26:23 UTC
Please note that after executing "foreman-rake katello:reindex --trace", I tried to sync same repo again and sync was completed successfully.

Please see the attachment how I cancel/kill the sync task.

Comment 6 Sachin Ghai 2015-09-09 09:26:57 UTC
Created attachment 1071655 [details]
cancelled sync task

Comment 7 Sachin Ghai 2015-09-09 09:27:47 UTC
As per comment 4, comment 5 and comment 6. Moving it to verified.

Comment 9 errata-xmlrpc 2015-09-15 07:22:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2015:1786

Comment 10 Paul Armstrong 2015-10-28 02:05:53 UTC
Confirmed in field. Encountered this problem on 6.1.2 due to a similar issue:

...
Re-indexing Katello::PuppetModule
Re-indexing Katello::Distribution
Re-indexing Katello::PackageGroup
Re-indexing Katello::Erratum
rake aborted!
undefined method `each' for #<Class:0x00000009749998>


upgrading to 6.1.3 resolved the problem.