Bug 1378913

Summary:

[upgrades] Auto-execute run in db:seed causes upgrade failures

Product:

Red Hat Satellite

Reporter:

Ivan Necas <inecas>

Component:

Tasks Plugin

Assignee:

Ivan Necas <inecas>

Status:

CLOSED ERRATA

QA Contact:

Lukas Pramuk <lpramuk>

Severity:

high

Docs Contact:

Priority:

medium

Version:

6.2.0

CC:

bbuckingham, inecas, jcallaha, lpramuk, mmccune, rbobek, sramacha, syangsao

Target Milestone:

Unspecified

Keywords:

Triaged

Target Release:

Unused

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

rubygem-foreman-tasks-0.7.14.9-1

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2016-10-26 12:26:54 UTC

Type:

---

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
foreman tasks stopped	none
foreman tasks started	none
sync tasks completed	none

Description Ivan Necas 2016-09-23 13:53:16 UTC

The auto-execute feature is meant to be run once the permanent
executor is starting to pick up tasks that were queed while no
executor was avaiable. However, we run executor also in db:seed (to
not be dependent on external executor at this phase). This can lead to
auto-execute to be run there as well, with errors like this:

Sequel::UniqueConstraintViolation: PG::Error: ERROR:  duplicate key value
violates unique constraint "dynflow_coordinator_records_pkey"
DETAIL:  Key (id, class)=(auto-execute,
Dynflow::Coordinator::AutoExecuteLock) already exists.

The solution should be not trying to auto-execute inside rake tasks

Comment 1 Ivan Necas 2016-09-23 13:53:21 UTC

Created from redmine issue http://projects.theforeman.org/issues/16666

Comment 2 Ivan Necas 2016-09-23 13:53:26 UTC

Upstream bug assigned to inecas

Comment 4 Bryan Kearney 2016-10-03 20:19:12 UTC

Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/16666 has been resolved.

Comment 7 Ivan Necas 2016-10-06 10:45:19 UTC

*** Bug 1374912 has been marked as a duplicate of this bug. ***

Comment 18 Ivan Necas 2016-10-13 08:07:11 UTC

Steps to verify (it doesn't have to be during upgrade, latest sat 6.2. is enough)

1. stop foreman-tasks
2. run some tasks, such as repo sync
3. see the task in pending state
4. run `foreman-rake db:seed`

Current result

the pending task switches to running/stopped during the db:seed

Expected result

the task is not touched when db:seed is started

the tasks starts being processed when foreman-tasks service is started again.

Comment 19 jcallaha 2016-10-13 13:59:11 UTC

Following the steps provided in #18, I had no issues with the db:seed operation as well as automatically resuming tasks to completion. See below and attached screenshots.

[root@rhsm-qe-1 workstation]# service foreman-tasks stop
Redirecting to /bin/systemctl stop  foreman-tasks.service
[root@rhsm-qe-1 workstation]# foreman-rake db:seed
Seeding /usr/share/foreman/db/seeds.d/03-auth_sources.rb
Seeding /usr/share/foreman/db/seeds.d/03-permissions.rb
Seeding /usr/share/foreman/db/seeds.d/03-roles.rb
Seeding /usr/share/foreman/db/seeds.d/04-admin.rb
Seeding /usr/share/foreman/db/seeds.d/05-taxonomies.rb
Seeding /usr/share/foreman/db/seeds.d/06-architectures.rb
Seeding /usr/share/foreman/db/seeds.d/07-provisioning_templates.rb
Seeding /usr/share/foreman/db/seeds.d/08-partition_tables.rb
Seeding /usr/share/foreman/db/seeds.d/10-installation_media.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.81/db/seeds.d/101-locations.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.81/db/seeds.d/102-organizations.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.81/db/seeds.d/103-provisioning_templates.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.81/db/seeds.d/104-proxy.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.81/db/seeds.d/106-mail_notifications.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.81/db/seeds.d/107-enable_dynflow.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.81/db/seeds.d/108-ensure_sync_notification.rb
Seeding /usr/share/foreman/db/seeds.d/11-smart_proxy_features.rb
Seeding /usr/share/foreman/db/seeds.d/13-compute_profiles.rb
Seeding /usr/share/foreman/db/seeds.d/15-bookmarks.rb
Seeding /usr/share/foreman/db/seeds.d/16-mail_notifications.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.14.9/db/seeds.d/20-foreman_tasks_permissions.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/redhat_access-1.0.13/db/seeds.d/200-update-insights-roles.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_bootdisk-6.1.0.3/db/seeds.d/50-bootdisk_templates.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_discovery-5.0.0.9/db/seeds.d/50_discovery_templates.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.14.9/db/seeds.d/60-dynflow_proxy_feature.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-0.3.0.12/db/seeds.d/60-ssh_proxy_feature.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_discovery-5.0.0.9/db/seeds.d/60_discovery_proxy_feature.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.14.9/db/seeds.d/61-foreman_tasks_bookmarks.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-0.3.0.12/db/seeds.d/70-job_templates.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.81/db/seeds.d/75-job_templates.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-0.3.0.12/db/seeds.d/90-bookmarks.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-0.1.30/db/seeds.d/990 - provisioning_templates.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_openscap-0.5.3.18/db/seeds.d/openscap_feature.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_openscap-0.5.3.18/db/seeds.d/openscap_policy_notification.rb
All seed files executed
[root@rhsm-qe-1 workstation]# service foreman-tasks start
Redirecting to /bin/systemctl start  foreman-tasks.service

Comment 20 jcallaha 2016-10-13 14:00:07 UTC

Created attachment 1210127 [details]
foreman tasks stopped

Comment 21 jcallaha 2016-10-13 14:00:31 UTC

Created attachment 1210128 [details]
foreman tasks started

Comment 22 jcallaha 2016-10-13 14:00:51 UTC

Created attachment 1210129 [details]
sync tasks completed

Comment 23 Lukas Pramuk 2016-10-13 14:19:50 UTC

VERIFIED.

@satellite-6.2.3-1.0.el7sat.noarch
tfm-rubygem-foreman-tasks-0.7.14.9-1.el7sat.noarch

by using manual reproducer described in comment#18:

...
4. Sync task remained pending after db:seed finished. 

After starting foreman-tasks again Sync task got running and finished successfully

Comment 25 errata-xmlrpc 2016-10-26 12:26:54 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:2108