Bug 1001609 - Org delete sporadically fails
Org delete sporadically fails
Status: CLOSED ERRATA
Product: Subscription Asset Manager
Classification: Red Hat
Component: katello (Show other bugs)
1.3
Unspecified Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: Tom McKay
Vitaly Kuznetsov
:
: 1004304 1006199 (view as bug list)
Depends On:
Blocks: sam13-tracker 881568
  Show dependency treegraph
 
Reported: 2013-08-27 07:42 EDT by Vitaly Kuznetsov
Modified: 2013-10-01 07:22 EDT (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-10-01 07:22:05 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
'reproduce.png' (156.68 KB, image/png)
2013-08-30 01:54 EDT, qianzhan
no flags Details

  None (edit)
Description Vitaly Kuznetsov 2013-08-27 07:42:00 EDT
Description of problem:
Sometimes removing an Org from WebUI fails with the following message:
Failed to delete organization 'testorg779E17'. Check notices for more details. 
PGError: ERROR: update or delete on table "organizations" violates foreign key constraint "notices_organization_id_fk" on table "notices" DETAIL: Key (id)=(2) is still referenced from table "notices". : COMMIT (ActiveRecord::InvalidForeignKey)

Version-Release number of selected component (if applicable):
# rpm -qa | grep -P 'katello|candlepin'
katello-glue-candlepin-1.4.3-9.el6sam_splice.noarch
candlepin-0.8.21-1.el6sam.noarch
candlepin-scl-rhino-1.7R3-1.el6_4.noarch
katello-common-1.4.3-9.el6sam_splice.noarch
katello-headpin-all-1.4.3-9.el6sam_splice.noarch
katello-candlepin-cert-key-pair-1.0-1.noarch
katello-glue-elasticsearch-1.4.3-9.el6sam_splice.noarch
candlepin-selinux-0.8.21-1.el6sam.noarch
candlepin-scl-runtime-1-5.el6_4.noarch
candlepin-scl-1-5.el6_4.noarch
katello-selinux-1.4.4-2.el6sat.noarch
katello-cli-common-1.4.3-8.el6sat.noarch
signo-katello-0.0.20-1.el6sat.noarch
katello-cli-1.4.3-8.el6sat.noarch
katello-configure-1.4.4-3.el6sat.noarch
katello-headpin-1.4.3-9.el6sam_splice.noarch
candlepin-tomcat6-0.8.21-1.el6sam.noarch
candlepin-scl-quartz-2.1.5-5.el6_4.noarch
katello-certs-tools-1.4.2-2.el6sat.noarch


How reproducible:
sometimes

Steps to Reproduce:
1. Try removing org in webui

Actual results:
Error message, org is not present in UI any more

Expected results:
Successful delete

Additional info:
Comment 2 sthirugn@redhat.com 2013-08-27 10:23:52 EDT
@Vitaly: Just curious to see if this is happening only with the default org - ACME_Corporation or with any org? Can you confirm.  I am trying to make sure that this is different from https://bugzilla.redhat.com/show_bug.cgi?id=881568
Comment 3 Vitaly Kuznetsov 2013-08-27 10:30:43 EDT
Actually I never tried removing default ACME_Corporation, it happened to me several times with non-default orgs (and you can see testorg779E17 with id=2 in Comment#0). 
IMHO it happens when there're undelivered (unread?) notifications for the org so it's not https://bugzilla.redhat.com/show_bug.cgi?id=881568.
Comment 4 mkovacik 2013-08-28 07:16:08 EDT
(In reply to Vitaly Kuznetsov from comment #3)
> Actually I never tried removing default ACME_Corporation, it happened to me
> several times with non-default orgs (and you can see testorg779E17 with id=2
> in Comment#0). 
> IMHO it happens when there're undelivered (unread?) notifications for the
> org so it's not https://bugzilla.redhat.com/show_bug.cgi?id=881568.

# I'm seeing the same log that is referenced in bug #881568 with a non-default org name:

[root@ec2-54-252-38-101 ~]# tail -25  /var/log/katello/delayed_job.log


# - - - - - -

2013-08-27T10:58:20-0400: [Worker(delayed_job host:ec2-54-252-38-101.ap-southeast-2.compute.amazonaws.com pid:13680)] OrganizationDestroyer#run failed with ActiveRecord::InvalidForeignKey: PGError: ERROR:  update or delete on tab
le "organizations" violates foreign key constraint "notices_organization_id_fk" on table "notices"
DETAIL:  Key (id)=(2) is still referenced from table "notices".
: COMMIT - 0 failed attempts
2013-08-27T10:58:20-0400: [Worker(delayed_job host:ec2-54-252-38-101.ap-southeast-2.compute.amazonaws.com pid:13680)] PERMANENTLY removing OrganizationDestroyer#run because of 1 consecutive failures.
PGError: ERROR:  update or delete on table "organizations" violates foreign key constraint "notices_organization_id_fk" on table "notices"
DETAIL:  Key (id)=(2) is still referenced from table "notices".
: COMMIT


# - - - - - -

/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/command.rb:78:in `times'
/opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/command.rb:78:in `daemonize'
script/delayed_job:5:in `<main>'
caused by: (PGError) ERROR:  update or delete on table "organizations" violates foreign key constraint "notices_organization_id_fk" on table "notices"
DETAIL:  Key (id)=(2) is still referenced from table "notices".

    ... skipped 77 lines

# which leads to an API call failure when trying to add the same org later on again:
katello.client.server.ServerRequestError: (422, {u'displayMessage': u'Validation failed: Name has already been taken, Label already exists (including organizations being deleted)', u'errors': [u'Validation failed: Name has alrea$y been taken, Label already exists (including organizations being deleted)']}, None)

# thus it seems this is a duplicate of bug #881568 indeed


# [root@ec2-54-252-38-101 ~]# rpm -q katello-common
# katello-common-1.4.3-8.el6sam_splice.noarch
Comment 5 sthirugn@redhat.com 2013-08-28 10:51:15 EDT
@mkovacik: Lets keep this bug open, as bug 881568 is specifically created for ACME_Corporation issue.
Comment 7 qianzhan 2013-08-30 01:54:36 EDT
Created attachment 792032 [details]
'reproduce.png'
Comment 8 qianzhan 2013-08-30 02:12:27 EDT
After tried many times, basically, I can always reproduce this bug with steps(1)~(3) in below part 2, and the following is the detailed info:

1. Version-Release number of SAM:
katello-candlepin-cert-key-pair-1.0-1.noarch
katello-certs-tools-1.4.2-2.el6sat.noarch
katello-cli-1.4.3-9.el6sat.noarch
katello-cli-common-1.4.3-9.el6sat.noarch
katello-common-1.4.3-11.el6sam_splice.noarch
katello-configure-1.4.4-3.el6sat.noarch
katello-glue-candlepin-1.4.3-11.el6sam_splice.noarch
katello-glue-elasticsearch-1.4.3-11.el6sam_splice.noarch
katello-headpin-1.4.3-11.el6sam_splice.noarch
katello-headpin-all-1.4.3-11.el6sam_splice.noarch
katello-selinux-1.4.4-2.el6sat.noarch
signo-katello-0.0.20-1.el6sat.noarch
candlepin-0.8.24-1.el6sam.noarch
candlepin-scl-1-5.el6_4.noarch
candlepin-scl-quartz-2.1.5-5.el6_4.noarch
candlepin-scl-rhino-1.7R3-1.el6_4.noarch
candlepin-scl-runtime-1-5.el6_4.noarch
candlepin-selinux-0.8.24-1.el6sam.noarch
candlepin-tomcat6-0.8.24-1.el6sam.noarch
thumbslug-0.0.32-1.el6sam.noarch
thumbslug-selinux-0.0.32-1.el6sam.noarch

2. How did I reproduce the bug:
(1) Login to SAM WEBUI.
(2) Create org named 'a1', and switch to the org 'a1' from the menu located on the upper left corner of the page, then upload a manifest for the new org successfully.
(3) Switch to org 'ACME_Corporation', and delete org 'a1', then create another org 'a2', the error message pops up (Please the attachment 'reproduce.png').

3. katello-debug info: See attachment 'katello-debug info reproduce'
Comment 9 Apurva Bhole 2013-08-30 13:20:16 EDT
Org delete works fine with the cli though. Folowing the same case as in #Comment8 above using cli:

1. org create --name newOrg
2. provider import_manifest --name "Red Hat" --file "manifest_refresh.zip" --org newOrg
3. org delete --name newOrg
4. org create --name newOrg1
5 org list

Version-Release:
candlepin-scl-quartz-2.1.5-5.el6_4.noarch
katello-glue-elasticsearch-1.4.3-11.el6sam_splice.noarch
candlepin-scl-runtime-1-5.el6_4.noarch
katello-cli-common-1.4.3-9.el6sat.noarch
candlepin-tomcat6-0.8.24-1.el6sam.noarch
katello-candlepin-cert-key-pair-1.0-1.noarch
candlepin-selinux-0.8.24-1.el6sam.noarch
katello-configure-1.4.4-3.el6sat.noarch
signo-katello-0.0.20-1.el6sat.noarch
katello-selinux-1.4.4-2.el6sat.noarch
candlepin-scl-rhino-1.7R3-1.el6_4.noarch
katello-cli-1.4.3-9.el6sat.noarch
katello-common-1.4.3-11.el6sam_splice.noarch
katello-headpin-1.4.3-11.el6sam_splice.noarch
candlepin-0.8.24-1.el6sam.noarch
katello-headpin-all-1.4.3-11.el6sam_splice.noarch
katello-certs-tools-1.4.2-2.el6sat.noarch
candlepin-scl-1-5.el6_4.noarch
katello-glue-candlepin-1.4.3-11.el6sam_splice.noarch
Comment 10 sthirugn@redhat.com 2013-09-05 15:12:57 EDT
I am able to reproduce this as per Comment 8.  This looks like a nasty bug with foreign key violation error while updating notices.  This needs fix for sam 1.3
Comment 12 Tom McKay 2013-09-09 13:03:33 EDT
Branch: refs/heads/master
  Home:   https://github.com/Katello/katello
  Commit: 3ab047fd7be7f73a4111f478cda6a74ce826946b
      https://github.com/Katello/katello/commit/3ab047fd7be7f73a4111f478cda6a74ce826946b
  Author: Tom McKay <thomasmckay@redhat.com>
  Date:   2013-09-09 (Mon, 09 Sep 2013)
Comment 13 Garik Khachikyan 2013-09-10 08:59:27 EDT
*** Bug 1006199 has been marked as a duplicate of this bug. ***
Comment 14 sthirugn@redhat.com 2013-09-10 09:41:17 EDT
Note to QE: please test bug 1006199 just to make sure we are good there.
Comment 16 Tazim Kolhar 2013-09-12 06:37:04 EDT
Tried with rpms : 
# rpm -qa | grep katello
katello-cli-1.4.3-10.el6sat.noarch
katello-headpin-1.4.3-12.el6sam_splice.noarch
katello-configure-1.4.4-4.el6sat.noarch
katello-glue-candlepin-1.4.3-12.el6sam_splice.noarch
katello-selinux-1.4.4-2.el6sat.noarch
katello-candlepin-cert-key-pair-1.0-1.noarch
katello-certs-tools-1.4.2-2.el6sat.noarch
katello-cli-common-1.4.3-10.el6sat.noarch
katello-glue-elasticsearch-1.4.3-12.el6sam_splice.noarch
katello-common-1.4.3-12.el6sam_splice.noarch
katello-headpin-all-1.4.3-12.el6sam_splice.noarch


Test Class : com.redhat.qe.katello.tests.cli.UserOrgDeleteTests
Test Method :test_deleteOrgFirst 


The test contains the below steps :

# headpin -u admin -p **** user create --username testuser --password redhat --email testuser@redhat.com
Successfully created user [ testuser ]

# headpin --username *** --password "admin" user assign_role --username "testuser" --role "Administrator" 
User 'testuser' assigned to role 'Administrator'

# headpin -u testuser -p **** org create --name torg
Successfully created org [ torg ]

# headpin -u testuser -p *** org delete --name torg
Successfully deleted org [ torg ]            

# headpin -u admin -p ***** user delete --username testuser
ERROR:  update or delete on table "users" violates foreign key constraint "task_statuses_user_id_fk" on table "task_statuses"
DETAIL:  Key (id)=(6) is still referenced from table "task_statuses".
Comment 17 sthirugn@redhat.com 2013-09-12 16:11:00 EDT
Heard that Vitaly Kuznetsov will be verifying this. Assigning the bug to Vitaly.
Comment 18 Vitaly Kuznetsov 2013-09-13 04:14:31 EDT
(In reply to sthirugn from comment #17)
> Heard that Vitaly Kuznetsov will be verifying this. Assigning the bug to
> Vitaly.

Np, verified with snapshot 5. Not hitting the issue any more and reproducer from Comment#8 fails to reproduce the issue.
Comment 19 Vitaly Kuznetsov 2013-09-13 04:20:51 EDT
Sorry, after selecting new org (a2 from Comment#8) I'm seeing the issue again :-(

Failed to delete organization 'a1'. Check notices for more details. 
PGError: ERROR: update or delete on table "notices" violates foreign key constraint "user_notices_notice_id_fk" on table "user_notices" DETAIL: Key (id)=(3) is still referenced from table "user_notices". : COMMIT (ActiveRecord::InvalidForeignKey)

The exact sequence is:
1) Create org a1
2) Switch to org a1
3) Upload manifest to a1, wait till upload succeed
4) Switch to ACME_Corporation
5) Go to Manage organizations
6) Delete org a1
7) Create org a2
8) Select org a2, go to Dashboard
9) See the issue

I've tried with Snapshot6 and bug is still there. Returning to ASSIGNED.
Comment 20 Tom McKay 2013-09-13 15:03:02 EDT
Branch: refs/heads/master
  Home:   https://github.com/Katello/katello
  Commit: a457c64ab8490fb00cb2a371ebe0d662d2e117cd
      https://github.com/Katello/katello/commit/a457c64ab8490fb00cb2a371ebe0d662d2e117cd
  Author: Tom McKay <thomasmckay@redhat.com>
  Date:   2013-09-13 (Fri, 13 Sep 2013)
Comment 21 Tom McKay 2013-09-17 08:14:27 EDT
*** Bug 1004304 has been marked as a duplicate of this bug. ***
Comment 22 Bryan Kearney 2013-09-18 12:07:17 EDT
moving to ON_QA, these are in Snap7.
Comment 23 sthirugn@redhat.com 2013-09-18 13:51:32 EDT
Failed.

Saw the following error right after Step 7 in Comment 19:

Failed to delete organization 'org1'. Check notices for more details.
PGError: ERROR: update or delete on table "notices" violates foreign key constraint "user_notices_notice_id_fk" on table "user_notices" DETAIL: Key (id)=(10) is still referenced from table "user_notices". : COMMIT (ActiveRecord::InvalidForeignKey) 


Version Tested: (SAM 1.3 Snap 7)
* candlepin-0.8.26-1.el6sam.noarch
* candlepin-scl-1-5.el6_4.noarch
* candlepin-scl-quartz-2.1.5-5.el6_4.noarch
* candlepin-scl-rhino-1.7R3-1.el6_4.noarch
* candlepin-scl-runtime-1-5.el6_4.noarch
* candlepin-selinux-0.8.26-1.el6sam.noarch
* candlepin-tomcat6-0.8.26-1.el6sam.noarch
* elasticsearch-0.19.9-8.el6sat.noarch
* katello-candlepin-cert-key-pair-1.0-1.noarch
* katello-certs-tools-1.4.2-2.el6sat.noarch
* katello-cli-1.4.3-11.el6sat.noarch
* katello-cli-common-1.4.3-11.el6sat.noarch
* katello-common-1.4.3-14.el6sam_splice.noarch
* katello-configure-1.4.4-4.el6sat.noarch
* katello-glue-candlepin-1.4.3-14.el6sam_splice.noarch
* katello-glue-elasticsearch-1.4.3-14.el6sam_splice.noarch
* katello-headpin-1.4.3-14.el6sam_splice.noarch
* katello-headpin-all-1.4.3-14.el6sam_splice.noarch
* katello-selinux-1.4.4-2.el6sat.noarch
* thumbslug-0.0.34-1.el6sam.noarch
* thumbslug-selinux-0.0.34-1.el6sam.noarch
Comment 24 Tom McKay 2013-09-19 09:13:40 EDT
Branch: refs/heads/master
  Home:   https://github.com/Katello/katello
  Commit: 3b794e7ace8041dc3d90d2271d01aa0350d2800c
      https://github.com/Katello/katello/commit/3b794e7ace8041dc3d90d2271d01aa0350d2800c
  Author: Tom McKay <thomasmckay@redhat.com>
  Date:   2013-09-19 (Thu, 19 Sep 2013)
Comment 25 Bryan Kearney 2013-09-20 15:14:19 EDT
Delivered in SNAP 7.1
Comment 26 sthirugn@redhat.com 2013-09-20 16:18:18 EDT
Verified.  But waiting for Vitaly to mark this verified.

Version Tested:
* candlepin-0.8.26-1.el6sam.noarch
* candlepin-cert-consumer-cloud-qe-19.idm.lab.bos.redhat.com-1.0-1.noarch
* candlepin-scl-1-5.el6_4.noarch
* candlepin-scl-quartz-2.1.5-5.el6_4.noarch
* candlepin-scl-rhino-1.7R3-1.el6_4.noarch
* candlepin-scl-runtime-1-5.el6_4.noarch
* candlepin-selinux-0.8.26-1.el6sam.noarch
* candlepin-tomcat6-0.8.26-1.el6sam.noarch
* elasticsearch-0.19.9-8.el6sat.noarch
* katello-candlepin-cert-key-pair-1.0-1.noarch
* katello-certs-tools-1.4.2-2.el6sat.noarch
* katello-cli-1.4.3-11.el6sat.noarch
* katello-cli-common-1.4.3-11.el6sat.noarch
* katello-common-1.4.3-15.el6sam_splice.noarch
* katello-configure-1.4.4-4.el6sat.noarch
* katello-glue-candlepin-1.4.3-15.el6sam_splice.noarch
* katello-glue-elasticsearch-1.4.3-15.el6sam_splice.noarch
* katello-headpin-1.4.3-15.el6sam_splice.noarch
* katello-headpin-all-1.4.3-15.el6sam_splice.noarch
* katello-selinux-1.4.4-2.el6sat.noarch
* thumbslug-0.0.34-1.el6sam.noarch
* thumbslug-selinux-0.0.34-1.el6sam.noarch
Comment 27 Vitaly Kuznetsov 2013-09-23 05:27:37 EDT
Confirm, Verified with Snapshot 7.1
Comment 29 errata-xmlrpc 2013-10-01 07:22:05 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2013-1390.html

Note You need to log in before you can comment on or make changes to this bug.