Hide Forgot
Description of problem: The delayed_jobs.log file is showing a stacktrace for my hourly sync for a custom product. The stacktrace shows up every hour. Version-Release number of selected component (if applicable): * apr-util-ldap-1.3.9-3.el6_0.1.x86_64 * candlepin-0.8.25-1.el6sam.noarch * candlepin-scl-1-5.el6_4.noarch * candlepin-scl-quartz-2.1.5-5.el6_4.noarch * candlepin-scl-rhino-1.7R3-1.el6_4.noarch * candlepin-scl-runtime-1-5.el6_4.noarch * candlepin-selinux-0.8.25-1.el6sam.noarch * candlepin-tomcat6-0.8.25-1.el6sam.noarch * elasticsearch-0.19.9-8.el6sat.noarch * foreman-1.3.0-18.el6sat.noarch * foreman-compute-1.3.0-18.el6sat.noarch * foreman-libvirt-1.3.0-18.el6sat.noarch * foreman-postgresql-1.3.0-18.el6sat.noarch * foreman-proxy-1.3.0-3.el6sat.noarch * katello-1.4.6-39.el6sat.noarch * katello-all-1.4.6-39.el6sat.noarch * katello-candlepin-cert-key-pair-1.0-1.noarch * katello-certs-tools-1.4.4-1.el6sat.noarch * katello-cli-1.4.3-24.el6sat.noarch * katello-cli-common-1.4.3-24.el6sat.noarch * katello-common-1.4.6-39.el6sat.noarch * katello-configure-1.4.7-5.el6sat.noarch * katello-configure-foreman-1.4.7-5.el6sat.noarch * katello-configure-foreman-proxy-1.4.7-5.el6sat.noarch * katello-foreman-all-1.4.6-39.el6sat.noarch * katello-glue-candlepin-1.4.6-39.el6sat.noarch * katello-glue-elasticsearch-1.4.6-39.el6sat.noarch * katello-glue-pulp-1.4.6-39.el6sat.noarch * katello-qpid-broker-key-pair-1.0-1.noarch * katello-qpid-client-key-pair-1.0-1.noarch * katello-selinux-1.4.4-4.el6sat.noarch * openldap-2.4.23-31.el6.x86_64 * pulp-katello-plugins-0.2-1.el6sat.noarch * pulp-nodes-common-2.3.0-0.22.beta.el6sat.noarch * pulp-nodes-parent-2.3.0-0.22.beta.el6sat.noarch * pulp-puppet-plugins-2.3.0-0.22.beta.el6sat.noarch * pulp-rpm-plugins-2.3.0-0.22.beta.el6sat.noarch * pulp-selinux-2.3.0-0.22.beta.el6sat.noarch * pulp-server-2.3.0-0.22.beta.el6sat.noarch * python-ldap-2.3.10-1.el6.x86_64 * ruby193-rubygem-ldap_fluff-0.2.2-2.el6sat.noarch * ruby193-rubygem-net-ldap-0.3.1-3.el6sat.noarch * ruby193-rubygem-runcible-1.0.7-1.el6sat.noarch * signo-0.0.22-2.el6sat.noarch * signo-katello-0.0.22-2.el6sat.noarch How reproducible: Steps to Reproduce: function kk() { KATELLO_PATH=/usr/bin/katello $KATELLO_PATH -u admin -p admin "$@" } # Orgs ## Simple Org kk org create --name "Katello Infrastructure" kk environment create --org "Katello Infrastructure" --name 'DEV' --prior Library kk environment create --org "Katello Infrastructure" --name 'QA' --prior 'DEV' kk environment create --org "Katello Infrastructure" --name 'GA' --prior 'QA' # Switch to $ORG Org kk client remember --option org --value "Katello Infrastructure" kk client remember --option env --value 'DEV' # Custom provider kk provider create --name="Google" wget https://dl-ssl.google.com/linux/linux_signing_key.pub kk gpg_key create --name 'google-gpg' --file linux_signing_key.pub # Google product kk product create --name="Chrome" --provider="Google" kk product update --name="Chrome" --gpgkey 'google-gpg' kk repo create --name="google-chrome-32bit" --product="Chrome" --url=http://dl.google.com/linux/chrome/rpm/stable/i386 --unprotected kk repo create --name="google-chrome-64bit" --product="Chrome" --url=http://dl.google.com/linux/chrome/rpm/stable/x86_64 --unprotected kk sync_plan create --interval hourly --date `date +"%Y-%m-%d"` --time `date -d "+20 minute" +"%H:%M:%S"` --name "Hourly" kk product set_plan --name "Chrome" --plan Hourly Actual results: ==> /var/log/katello/delayed_job.log <== 2013-10-18T09:55:01-0400: [Worker(delayed_job host:cloud-qe-11.idm.lab.bos.redhat.com pid:16225)] Repository#after_sync failed with NoMethodError: undefined method `error_details' for #<ActiveSupport::HashWithIndifferentAccess:0x0000001525ec10> - 0 failed attempts 2013-10-18T09:55:01-0400: [Worker(delayed_job host:cloud-qe-11.idm.lab.bos.redhat.com pid:16225)] PERMANENTLY removing Repository#after_sync because of 1 consecutive failures. undefined method `error_details' for #<ActiveSupport::HashWithIndifferentAccess:0x0000001525ec10> Expected results: Additional info: 2013-10-18T07:55:00-0400: [Worker(delayed_job host:cloud-qe-11.idm.lab.bos.redhat.com pid:16225)] Repository#after_sync failed with NoMethodError: undefined method `error_details' for #<ActiveSupport::HashWithIndifferentAccess:0x00000009ec1ce8> - 0 failed attempts 2013-10-18T07:55:00-0400: [Worker(delayed_job host:cloud-qe-11.idm.lab.bos.redhat.com pid:16225)] PERMANENTLY removing Repository#after_sync because of 1 consecutive failures. undefined method `error_details' for #<ActiveSupport::HashWithIndifferentAccess:0x00000009ec1ce8> /usr/share/katello/app/models/glue/pulp/repo.rb:478:in `handle_sync_complete_task' /usr/share/katello/app/models/repository.rb:187:in `after_sync' /usr/share/katello/app/models/async_operation.rb:57:in `perform' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/backend/base.rb:94:in `block in invoke_job' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:60:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:60:in `block in initialize' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:65:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:65:in `execute' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:38:in `run_callbacks' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/backend/base.rb:91:in `invoke_job' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/worker.rb:178:in `block (2 levels) in run' /opt/rh/ruby193/root/usr/share/ruby/timeout.rb:69:in `timeout' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/worker.rb:178:in `block in run' /opt/rh/ruby193/root/usr/share/ruby/benchmark.rb:295:in `realtime' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/worker.rb:177:in `run' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/worker.rb:234:in `block in reserve_and_run_one_job' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:60:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:60:in `block in initialize' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:65:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:65:in `execute' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:38:in `run_callbacks' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/worker.rb:234:in `reserve_and_run_one_job' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/worker.rb:162:in `block in work_off' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/worker.rb:161:in `times' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/worker.rb:161:in `work_off' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/worker.rb:133:in `block (4 levels) in start' /opt/rh/ruby193/root/usr/share/ruby/benchmark.rb:295:in `realtime' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/worker.rb:132:in `block (3 levels) in start' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:60:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:60:in `block in initialize' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:65:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:65:in `execute' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:38:in `run_callbacks' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/worker.rb:129:in `block (2 levels) in start' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/worker.rb:128:in `loop' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/worker.rb:128:in `block in start' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/plugins/clear_locks.rb:7:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/plugins/clear_locks.rb:7:in `block (2 levels) in <class:ClearLocks>' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:78:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:78:in `block (2 levels) in add' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:60:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:60:in `block in initialize' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:78:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:78:in `block in add' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:65:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:65:in `execute' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:38:in `run_callbacks' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/worker.rb:127:in `start' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/command.rb:101:in `run' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/command.rb:89:in `block in run_process' /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons/application.rb:249:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons/application.rb:249:in `block in start_proc' /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons/daemonize.rb:197:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons/daemonize.rb:197:in `call_as_daemon' /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons/application.rb:253:in `start_proc' /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons/application.rb:293:in `start' /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons/controller.rb:70:in `run' /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons.rb:195:in `block in run_proc' /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons/cmdline.rb:109:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons/cmdline.rb:109:in `catch_exceptions' /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons.rb:194:in `run_proc' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/command.rb:87:in `run_process' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/command.rb:80:in `block in daemonize' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/command.rb:78:in `times' /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/command.rb:78:in `daemonize' script/delayed_job:5:in `<main>'
Since this was created, we've gone through a lot of changes upstream (e.g. new releases of pulp, dynflowing of syncs..etc). With those changes, we won't be able to replicate this issue. Going to close this BZ for now and if we find issues with sync, new ones can be created.