Bug 2081560 - ForeignKeyViolation Error with docker_meta_tags
Summary: ForeignKeyViolation Error with docker_meta_tags
Keywords:
Status: POST
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Repositories
Version: 6.9.9
Hardware: Unspecified
OS: Unspecified
high
high with 1 vote vote
Target Milestone: Unspecified
Assignee: Ian Ballou
QA Contact: Cole Higgins
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-05-04 01:08 UTC by Yoga
Modified: 2022-09-28 00:23 UTC (History)
19 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)
Hotfix RPM for Satellite 6.9.9 (9.32 MB, application/x-rpm)
2022-07-27 20:44 UTC, wclark
no flags Details
Updated Hotfix RPM for Satellite 6.9.9 (9.32 MB, application/x-rpm)
2022-09-07 20:55 UTC, wclark
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 35233 0 High Closed ForeignKeyViolation Error with docker_meta_tags 2022-09-01 20:07:14 UTC
Red Hat Knowledge Base (Solution) 6960392 0 None None None 2022-05-23 13:33:16 UTC

Description Yoga 2022-05-04 01:08:24 UTC
Description of problem:
When upgrading from 6.9.9 to 6.10 we are facing the issue "ActiveRecord::InvalidForeignKey: PG::ForeignKeyViolation: ERROR:  update or delete on table "katello_docker_meta_tags" violates foreign key constraint "fk_rails_af711a226d" on table "katello_repository_docker_meta_tags"

Version-Release number of selected component (if applicable):
6.9.9


Actual results:

Switching specified content over to pulp 3                            [FAIL]
Failed executing foreman-rake katello:pulp3_content_switchover, exit status 1:
 rake aborted!
ActiveRecord::InvalidForeignKey: PG::ForeignKeyViolation: ERROR:  update or delete on table "katello_docker_meta_tags" violates foreign key constraint "fk_rails_af711a226d" on table "katello_repository_docker_meta_tags"

Expected results:

The upgrade should be successful


Additional info:

We already have a bug report with same issue https://bugzilla.redhat.com/show_bug.cgi?id=1946339 but the fix was in 6.9.2.The customer is already in 6.9.9 and hence the new bug

Comment 24 Ian Ballou 2022-07-15 18:47:19 UTC
Created redmine issue https://projects.theforeman.org/issues/35233 from this bug

Comment 29 wclark 2022-07-27 20:44:59 UTC
Created attachment 1899805 [details]
Hotfix RPM for Satellite 6.9.9

INSTALL INSTRUCTIONS:

1. Take a complete backup or snapshot of Satellite 6.9.9 server

2. Download the hotfix RPM for Satellite 6.9.9 attached to this BZ and copy it to Satellite server

3. # yum install ./tfm-rubygem-katello-3.18.1.53-2.HOTFIXRHBZ2081560.el7sat.noarch.rpm --disableplugin=foreman-protector

4. # satellite-maintain service restart

AFTER INSTALLATION:

The new logic to correct docker meta tags will run automatically during the switchover to pulp3 content. Users who instead want to perform the cleanup immediately without yet running the full content switchover can do so using the command `foreman-rake katello:correct_docker_meta_tags`

Comment 31 Vincent S. Cojot 2022-08-03 14:00:15 UTC
Any chance to see it in a 6.9.z for customers who wish to stay on 6.9.z a little longer due to 6.10.z being disruptive to RHOSP workflows (port 5000)?
Thank you,

Comment 32 Caruchamiuller 2022-08-12 20:47:41 UTC
(In reply to wclark from comment #29)
> Created attachment 1899805 [details]
> Hotfix RPM for Satellite 6.9.9
> 
> INSTALL INSTRUCTIONS:
> 
> 1. Take a complete backup or snapshot of Satellite 6.9.9 server
> 
> 2. Download the hotfix RPM for Satellite 6.9.9 attached to this BZ and copy
> it to Satellite server
> 
> 3. # yum install
> ./tfm-rubygem-katello-3.18.1.53-2.HOTFIXRHBZ2081560.el7sat.noarch.rpm
> --disableplugin=foreman-protector
> 
> 4. # satellite-maintain service restart
> 
> AFTER INSTALLATION:
> 
> The new logic to correct docker meta tags will run automatically during the
> switchover to pulp3 content. Users who instead want to perform the cleanup
> immediately without yet running the full content switchover can do so using
> the command `foreman-rake katello:correct_docker_meta_tags`


@wclark Hi. Nice to greet you.
I am on a Satellite 6.9.9 installation and I got the same error as at the beginning of the thread.

I did the steps that you mentioned and ran the command satellite-maintain upgrade run --target-version 6.10 again.

I got the same errors again:

PG::ForeignKeyViolation: ERROR: update or delete on table "katello_docker_meta_tags" violates foreign key constraint "fk_rails_af711a226d" on table "katello_repository_docker_meta_tags"
DETAIL: Key (id)=(62183) is still referenced from table "katello_repository_docker_meta_tags".

Could you tell me if I need to perform any additional steps?

When I ran the upgrade again the satellite server was not available. I don't know if I did the execution in the correct way.

How can I check if the hotfix already solves the problem without re-running the upgrade to 6.10?

Sorry for the questions but I am new to Satellite and we have been trying for months to upgrade to 6.10 without success.

Regards

Comment 33 Ian Ballou 2022-08-15 13:35:55 UTC
Hi @Caruchamiuller,

In your production logs, do you see "CONTENT_SWITCHOVER - correct_docker_meta_tags" and then a duration?

Did you try running `foreman-rake katello:correct_docker_meta_tags`? I'd recommend trying that before proceeding with the upgrade.

One way to help tell if the issue has been cleaned up would be by running the following:

```
::Katello::DockerTag.all.select { (t.schema1_meta_tag.present? && t.docker_taggable.schema_version == 2) || (t.schema2_meta_tag.present? && t.docker_taggable.schema_version == 1) }
```

If it returns something, then there is still incorrect data.


If you see the log line above in your satellite logs (after the upgrade attempt) and have run the rake task I recommended without success, we'd probably want to discuss looking at a satellite-maintain backup of your system.

Comment 34 Caruchamiuller 2022-08-15 15:39:15 UTC
@Ian Ballou, Hi
I am answering between the lines

> In your production logs, do you see "CONTENT_SWITCHOVER -
> correct_docker_meta_tags" and then a duration?

Yes, after applying the fix it can be seen in the logs

2022-08-11T03:34:40 [I|app|] CONTENT_SWITCHOVER - correct_docker_meta_tags duration=10631159.73
> 
> Did you try running `foreman-rake katello:correct_docker_meta_tags`? I'd
> recommend trying that before proceeding with the upgrade.

Ok, I'll keep it in mind

> 
> One way to help tell if the issue has been cleaned up would be by running
> the following:
> 
> ```
> ::Katello::DockerTag.all.select { (t.schema1_meta_tag.present? &&
> t.docker_taggable.schema_version == 2) || (t.schema2_meta_tag.present? &&
> t.docker_taggable.schema_version == 1) }
> ```

Excuse my ignorance but I run the command as you put it but it returns a syntax error

[root@satellite]# Katello::DockerTag.all.select { (t.schema1_meta_tag.present? && t.docker_taggable.schema_version == 2) || (t.schema2_meta_tag.present? && t.docker_taggable.schema_version == 1) }
bash: syntax error near unexpected token `('

What is the correct way to execute the command?

Regards.

Comment 35 Sayan Das 2022-08-15 15:49:31 UTC
You will need to go inside "foreman-rake console" first and then run the command.

or

# echo "Katello::DockerTag.all.select { (t.schema1_meta_tag.present? && t.docker_taggable.schema_version == 2) || (t.schema2_meta_tag.present? && t.docker_taggable.schema_version == 1) }" | foreman-rake console

Comment 36 Ian Ballou 2022-08-15 15:52:45 UTC
Thanks Sayan.

Also, since you're new to the Satellite console, please note that it's an administrative environment where data inconsistencies can be created if the wrong command is run by accident.  The command I sent is a query that causes no modification to your Satellite.

Comment 37 Caruchamiuller 2022-08-15 16:12:11 UTC
Thanks to both of you.
Without having run foreman-rake katello:correct_docker_meta_tags the result is as follows:

Loading production environment (Rails 6.0.3.4)
Switch to inspect mode.
Katello::DockerTag.all.select { (t.schema1_meta_tag.present? && t.docker_taggable.schema_version == 2) || (t.schema2_meta_tag.present? && t.docker_taggable.schema_version == 1) }
Traceback (most recent call last):
        3: from lib/tasks/console.rake:5:in `block in <top (required)>'
        2: from (irb):1
        1: from (irb):1:in `block in irb_binding'
NameError (undefined local variable or method `t' for main:Object)

With the hotfix applied is it necessary to rerun the satellite-maintain content prepare command?
Because I did not rerun satellite-maintain content prepare after applying the hotfix. 
Directly run satellite-maintain upgrade run --target-version 6.10

Comment 38 Ian Ballou 2022-08-15 16:22:22 UTC
Apologies @Caruchamiuller, try this instead:

```
Katello::DockerTag.all.select { |t| (t.schema1_meta_tag.present? && t.docker_taggable.schema_version == 2) || (t.schema2_meta_tag.present? && t.docker_taggable.schema_version == 1) }
```

`satellite-maintain content prepare` shouldn't be required again after applying the hotfix.

And the Satellite services were restarted after applying the hotfix?

Comment 39 Caruchamiuller 2022-08-15 17:30:09 UTC
Yes, after applying the fix the satellite services were restarted.
Then run `satellite-maintain upgrade upgrade run --target-version 6.10` again.
To fail again with the same error

Comment 40 Caruchamiuller 2022-08-15 18:07:05 UTC
@iballou 

Does the execution of Katello::DockerTag.all.select ... has any termination output?

Or is it a task that has a delay?

Because it is a few minutes ago without returning anything.

[root@satellite foreman]# echo "Katello::DockerTag.all.select { |t| (t.schema1_meta_tag.present? && t.docker_taggable.schema_version == 2) || (t.schema2_meta_tag.present? && t.docker_taggable.schema_version == 1) }" | foreman-rake console
Loading production environment (Rails 6.0.3.4)
Switch to inspect mode.
Katello::DockerTag.all.select { |t| (t.schema1_meta_tag.present? && t.docker_taggable.schema_version == 2) || (t.schema2_meta_tag.present? && t.docker_taggable.schema_version == 1) }

Comment 41 Ian Ballou 2022-08-15 19:21:05 UTC
Hi Caruchamiuller, it is a task with a delay indeed. It scales with the number of docker tags that you have in your environment.  It's a DB query that's run against all of your docker tags.

If it's taking too long, we could have it break at the first corrupt unit with the following modified command:

Katello::DockerTag.all.find { |t| (t.schema1_meta_tag.present? && t.docker_taggable.schema_version == 2) || (t.schema2_meta_tag.present? && t.docker_taggable.schema_version == 1) }

Comment 43 Caruchamiuller 2022-08-15 19:33:45 UTC
@iballou Thanks

Yes, it took a long time but I ended up with a lot of lines. I understand that these are all the corrupted files that are still on the system.

Does `foreman-rake katello:correct_docker_meta_tags` have any impact on the system still running version 6.9.9?

I am looking into running it to see what output it gives. But first I would like to know if it has any impact.

Regards

Comment 44 Ian Ballou 2022-08-15 19:59:52 UTC
@Caruchamiuller,

`foreman-rake katello:correct_docker_meta_tags` makes adjustments to your Docker repositories in the backend that are destructive. Essentially, corrupt units are removed and then recreated. In the meantime while the task is running, your affected repositories are in a temporary unknown state and shouldn't be synced, deleted, published in content views, etc.

The task itself will not give any meaningful output, but it would be helpful to run the command that I gave you to check again for corrupt units.

If you're still seeing corrupt units, we would need to debug further in the Satellite database to determine why the updated code isn't correcting the issue.

Comment 47 Caruchamiuller 2022-08-22 14:16:42 UTC
@iballou 

Hi,
I ran the command `foreman-rake katello:correct_docker_meta_tags`.

After that, I ran again 
Katello::DockerTag.all.select { |t| (t.schema1_meta_tag.present? && t.docker_taggable.schema_version == 2) || (t.schema2_meta_tag.present? && t.docker_taggable.schema_version == 1) }

The output of the above command returned nothing.

I ran the Satellite upgrade again but it failed again with the same error. The only difference is that I changed the Key ID.

[2022-08-19 19:32:29-0300 #6705] ERROR -- : Failed executing foreman-rake katello:pulp3_content_switchover, exit status 1:
 rake aborted!
ActiveRecord::InvalidForeignKey: PG::ForeignKeyViolation: ERROR:  update or delete on table "katello_docker_meta_tags" violates foreign key constraint "fk_rails_af711a226d" on table "katello_repository_docker_meta_tags"
DETAIL:  Key (id)=(88210) is still referenced from table "katello_repository_docker_meta_tags"

Comment 53 Bryan Kearney 2022-08-29 16:04:47 UTC
Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/35233 has been resolved.

Comment 54 Ian Ballou 2022-08-29 18:08:10 UTC
Automation is going to keep moving it to POST... please ignore it while I continue looking into the latest issue.

Comment 64 Weinan 2022-09-06 03:06:14 UTC
hi all, 

I am TAM of Services Australia. Services Australia is our Australian Federal Government customer - biggest customer at APAC. 

Could we please take this as priority? Our customer has wait for this get solved for a long time, stack and shift team also are impacted. They have experienced with corrupted container images for a log time, this will be fixed in 6.10. 

they spent 4 months to upgrade the Satellite servers case stuck the bug for long time, current customer temperature is red. 

Could we please solve this bug as first priority? Thank you. 

Weinan Zhang 
Technical Account Manager - Services Australia 
Red Hat

Comment 70 wclark 2022-09-07 20:55:54 UTC
Created attachment 1910326 [details]
Updated Hotfix RPM for Satellite 6.9.9

INSTALL INSTRUCTIONS:

1. Take a complete backup or snapshot of Satellite 6.9.9 server

2. Download the updated hotfix RPM for Satellite 6.9.9 attached to this BZ and copy it to Satellite server

3. # yum install ./tfm-rubygem-katello-3.18.1.53-3.HOTFIXRHBZ2081560.el7sat.noarch.rpm --disableplugin=foreman-protector

4. # satellite-maintain service restart

AFTER INSTALLATION:

The new logic to correct docker meta tags will run automatically during the switchover to pulp3 content. Users who instead want to perform the cleanup immediately without yet running the full content switchover can do so using the command `foreman-rake katello:correct_docker_meta_tags`


Note You need to log in before you can comment on or make changes to this bug.