Bug 1020931

Summary: Scheduled sync of product fails with stacktrace: undefined method `error_details'
Product: Red Hat Satellite Reporter: Og Maciel <omaciel>
Component: Content ManagementAssignee: Katello Bug Bin <katello-bugs>
Status: CLOSED UPSTREAM QA Contact: Katello QA List <katello-qa-list>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 6.0.2CC: bbuckingham, mmccune
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-07-16 19:52:01 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Og Maciel 2013-10-18 14:27:06 UTC
Description of problem:

The delayed_jobs.log file is showing a stacktrace for my hourly sync for a custom product. The stacktrace shows up every hour.

Version-Release number of selected component (if applicable):

* apr-util-ldap-1.3.9-3.el6_0.1.x86_64
* candlepin-0.8.25-1.el6sam.noarch
* candlepin-scl-1-5.el6_4.noarch
* candlepin-scl-quartz-2.1.5-5.el6_4.noarch
* candlepin-scl-rhino-1.7R3-1.el6_4.noarch
* candlepin-scl-runtime-1-5.el6_4.noarch
* candlepin-selinux-0.8.25-1.el6sam.noarch
* candlepin-tomcat6-0.8.25-1.el6sam.noarch
* elasticsearch-0.19.9-8.el6sat.noarch
* foreman-1.3.0-18.el6sat.noarch
* foreman-compute-1.3.0-18.el6sat.noarch
* foreman-libvirt-1.3.0-18.el6sat.noarch
* foreman-postgresql-1.3.0-18.el6sat.noarch
* foreman-proxy-1.3.0-3.el6sat.noarch
* katello-1.4.6-39.el6sat.noarch
* katello-all-1.4.6-39.el6sat.noarch
* katello-candlepin-cert-key-pair-1.0-1.noarch
* katello-certs-tools-1.4.4-1.el6sat.noarch
* katello-cli-1.4.3-24.el6sat.noarch
* katello-cli-common-1.4.3-24.el6sat.noarch
* katello-common-1.4.6-39.el6sat.noarch
* katello-configure-1.4.7-5.el6sat.noarch
* katello-configure-foreman-1.4.7-5.el6sat.noarch
* katello-configure-foreman-proxy-1.4.7-5.el6sat.noarch
* katello-foreman-all-1.4.6-39.el6sat.noarch
* katello-glue-candlepin-1.4.6-39.el6sat.noarch
* katello-glue-elasticsearch-1.4.6-39.el6sat.noarch
* katello-glue-pulp-1.4.6-39.el6sat.noarch
* katello-qpid-broker-key-pair-1.0-1.noarch
* katello-qpid-client-key-pair-1.0-1.noarch
* katello-selinux-1.4.4-4.el6sat.noarch
* openldap-2.4.23-31.el6.x86_64
* pulp-katello-plugins-0.2-1.el6sat.noarch
* pulp-nodes-common-2.3.0-0.22.beta.el6sat.noarch
* pulp-nodes-parent-2.3.0-0.22.beta.el6sat.noarch
* pulp-puppet-plugins-2.3.0-0.22.beta.el6sat.noarch
* pulp-rpm-plugins-2.3.0-0.22.beta.el6sat.noarch
* pulp-selinux-2.3.0-0.22.beta.el6sat.noarch
* pulp-server-2.3.0-0.22.beta.el6sat.noarch
* python-ldap-2.3.10-1.el6.x86_64
* ruby193-rubygem-ldap_fluff-0.2.2-2.el6sat.noarch
* ruby193-rubygem-net-ldap-0.3.1-3.el6sat.noarch
* ruby193-rubygem-runcible-1.0.7-1.el6sat.noarch
* signo-0.0.22-2.el6sat.noarch
* signo-katello-0.0.22-2.el6sat.noarch

How reproducible:


Steps to Reproduce:

function kk() {
  KATELLO_PATH=/usr/bin/katello
  $KATELLO_PATH -u admin -p admin "$@"
}

# Orgs
## Simple Org
kk org create --name "Katello Infrastructure"
kk environment create --org "Katello Infrastructure" --name 'DEV' --prior Library
kk environment create --org "Katello Infrastructure" --name 'QA' --prior 'DEV'
kk environment create --org "Katello Infrastructure" --name 'GA' --prior 'QA'

# Switch to $ORG Org
kk client remember --option org --value "Katello Infrastructure"
kk client remember --option env --value 'DEV'

# Custom provider
kk provider create --name="Google"
wget https://dl-ssl.google.com/linux/linux_signing_key.pub
kk gpg_key create --name 'google-gpg' --file linux_signing_key.pub
# Google product
kk product create --name="Chrome" --provider="Google"
kk product update --name="Chrome" --gpgkey 'google-gpg'
kk repo create --name="google-chrome-32bit" --product="Chrome" --url=http://dl.google.com/linux/chrome/rpm/stable/i386 --unprotected
kk repo create --name="google-chrome-64bit" --product="Chrome" --url=http://dl.google.com/linux/chrome/rpm/stable/x86_64 --unprotected

kk sync_plan create --interval hourly --date `date +"%Y-%m-%d"` --time `date -d "+20 minute" +"%H:%M:%S"` --name "Hourly"
kk product set_plan --name "Chrome" --plan Hourly


Actual results:

==> /var/log/katello/delayed_job.log <==
2013-10-18T09:55:01-0400: [Worker(delayed_job host:cloud-qe-11.idm.lab.bos.redhat.com pid:16225)] Repository#after_sync failed with NoMethodError: undefined method `error_details' for #<ActiveSupport::HashWithIndifferentAccess:0x0000001525ec10> - 0 failed attempts
2013-10-18T09:55:01-0400: [Worker(delayed_job host:cloud-qe-11.idm.lab.bos.redhat.com pid:16225)] PERMANENTLY removing Repository#after_sync because of 1 consecutive failures.
undefined method `error_details' for #<ActiveSupport::HashWithIndifferentAccess:0x0000001525ec10>

Expected results:


Additional info:

2013-10-18T07:55:00-0400: [Worker(delayed_job host:cloud-qe-11.idm.lab.bos.redhat.com pid:16225)] Repository#after_sync failed with NoMethodError: undefined method `error_details' for #<ActiveSupport::HashWithIndifferentAccess:0x00000009ec1ce8> - 0 failed attempts
2013-10-18T07:55:00-0400: [Worker(delayed_job host:cloud-qe-11.idm.lab.bos.redhat.com pid:16225)] PERMANENTLY removing Repository#after_sync because of 1 consecutive failures.
undefined method `error_details' for #<ActiveSupport::HashWithIndifferentAccess:0x00000009ec1ce8>
/usr/share/katello/app/models/glue/pulp/repo.rb:478:in `handle_sync_complete_task'
/usr/share/katello/app/models/repository.rb:187:in `after_sync'
/usr/share/katello/app/models/async_operation.rb:57:in `perform'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/backend/base.rb:94:in `block in invoke_job'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:60:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:60:in `block in initialize'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:65:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:65:in `execute'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:38:in `run_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/backend/base.rb:91:in `invoke_job'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/worker.rb:178:in `block (2 levels) in run'
/opt/rh/ruby193/root/usr/share/ruby/timeout.rb:69:in `timeout'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/worker.rb:178:in `block in run'
/opt/rh/ruby193/root/usr/share/ruby/benchmark.rb:295:in `realtime'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/worker.rb:177:in `run'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/worker.rb:234:in `block in reserve_and_run_one_job'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:60:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:60:in `block in initialize'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:65:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:65:in `execute'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:38:in `run_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/worker.rb:234:in `reserve_and_run_one_job'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/worker.rb:162:in `block in work_off'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/worker.rb:161:in `times'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/worker.rb:161:in `work_off'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/worker.rb:133:in `block (4 levels) in start'
/opt/rh/ruby193/root/usr/share/ruby/benchmark.rb:295:in `realtime'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/worker.rb:132:in `block (3 levels) in start'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:60:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:60:in `block in initialize'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:65:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:65:in `execute'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:38:in `run_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/worker.rb:129:in `block (2 levels) in start'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/worker.rb:128:in `loop'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/worker.rb:128:in `block in start'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/plugins/clear_locks.rb:7:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/plugins/clear_locks.rb:7:in `block (2 levels) in <class:ClearLocks>'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:78:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:78:in `block (2 levels) in add'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:60:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:60:in `block in initialize'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:78:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:78:in `block in add'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:65:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:65:in `execute'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:38:in `run_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/worker.rb:127:in `start'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/command.rb:101:in `run'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/command.rb:89:in `block in run_process'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons/application.rb:249:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons/application.rb:249:in `block in start_proc'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons/daemonize.rb:197:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons/daemonize.rb:197:in `call_as_daemon'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons/application.rb:253:in `start_proc'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons/application.rb:293:in `start'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons/controller.rb:70:in `run'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons.rb:195:in `block in run_proc'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons/cmdline.rb:109:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons/cmdline.rb:109:in `catch_exceptions'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons.rb:194:in `run_proc'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/command.rb:87:in `run_process'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/command.rb:80:in `block in daemonize'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/command.rb:78:in `times'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/command.rb:78:in `daemonize'
script/delayed_job:5:in `<main>'

Comment 1 Brad Buckingham 2014-07-16 19:52:01 UTC
Since this was created, we've gone through a lot of changes upstream (e.g. new releases of pulp, dynflowing of syncs..etc).  With those changes, we won't be able to replicate this issue.  Going to close this BZ for now and if we find issues with sync, new ones can be created.