Created redmine issue https://projects.theforeman.org/issues/35233 from this bug

Created attachment 1899805 [details]
Hotfix RPM for Satellite 6.9.9

INSTALL INSTRUCTIONS:

1. Take a complete backup or snapshot of Satellite 6.9.9 server

2. Download the hotfix RPM for Satellite 6.9.9 attached to this BZ and copy it to Satellite server

3. # yum install ./tfm-rubygem-katello-3.18.1.53-2.HOTFIXRHBZ2081560.el7sat.noarch.rpm --disableplugin=foreman-protector

4. # satellite-maintain service restart

AFTER INSTALLATION:

The new logic to correct docker meta tags will run automatically during the switchover to pulp3 content. Users who instead want to perform the cleanup immediately without yet running the full content switchover can do so using the command `foreman-rake katello:correct_docker_meta_tags`

Any chance to see it in a 6.9.z for customers who wish to stay on 6.9.z a little longer due to 6.10.z being disruptive to RHOSP workflows (port 5000)?
Thank you,

(In reply to wclark from comment #29)
> Created attachment 1899805 [details]
> Hotfix RPM for Satellite 6.9.9
> 
> INSTALL INSTRUCTIONS:
> 
> 1. Take a complete backup or snapshot of Satellite 6.9.9 server
> 
> 2. Download the hotfix RPM for Satellite 6.9.9 attached to this BZ and copy
> it to Satellite server
> 
> 3. # yum install
> ./tfm-rubygem-katello-3.18.1.53-2.HOTFIXRHBZ2081560.el7sat.noarch.rpm
> --disableplugin=foreman-protector
> 
> 4. # satellite-maintain service restart
> 
> AFTER INSTALLATION:
> 
> The new logic to correct docker meta tags will run automatically during the
> switchover to pulp3 content. Users who instead want to perform the cleanup
> immediately without yet running the full content switchover can do so using
> the command `foreman-rake katello:correct_docker_meta_tags`


@wclark Hi. Nice to greet you.
I am on a Satellite 6.9.9 installation and I got the same error as at the beginning of the thread.

I did the steps that you mentioned and ran the command satellite-maintain upgrade run --target-version 6.10 again.

I got the same errors again:

PG::ForeignKeyViolation: ERROR: update or delete on table "katello_docker_meta_tags" violates foreign key constraint "fk_rails_af711a226d" on table "katello_repository_docker_meta_tags"
DETAIL: Key (id)=(62183) is still referenced from table "katello_repository_docker_meta_tags".

Could you tell me if I need to perform any additional steps?

When I ran the upgrade again the satellite server was not available. I don't know if I did the execution in the correct way.

How can I check if the hotfix already solves the problem without re-running the upgrade to 6.10?

Sorry for the questions but I am new to Satellite and we have been trying for months to upgrade to 6.10 without success.

Regards

Hi @Caruchamiuller,

In your production logs, do you see "CONTENT_SWITCHOVER - correct_docker_meta_tags" and then a duration?

Did you try running `foreman-rake katello:correct_docker_meta_tags`? I'd recommend trying that before proceeding with the upgrade.

One way to help tell if the issue has been cleaned up would be by running the following:

```
::Katello::DockerTag.all.select { (t.schema1_meta_tag.present? && t.docker_taggable.schema_version == 2) || (t.schema2_meta_tag.present? && t.docker_taggable.schema_version == 1) }
```

If it returns something, then there is still incorrect data.


If you see the log line above in your satellite logs (after the upgrade attempt) and have run the rake task I recommended without success, we'd probably want to discuss looking at a satellite-maintain backup of your system.

@Ian Ballou, Hi
I am answering between the lines

> In your production logs, do you see "CONTENT_SWITCHOVER -
> correct_docker_meta_tags" and then a duration?

Yes, after applying the fix it can be seen in the logs

2022-08-11T03:34:40 [I|app|] CONTENT_SWITCHOVER - correct_docker_meta_tags duration=10631159.73
> 
> Did you try running `foreman-rake katello:correct_docker_meta_tags`? I'd
> recommend trying that before proceeding with the upgrade.

Ok, I'll keep it in mind

> 
> One way to help tell if the issue has been cleaned up would be by running
> the following:
> 
> ```
> ::Katello::DockerTag.all.select { (t.schema1_meta_tag.present? &&
> t.docker_taggable.schema_version == 2) || (t.schema2_meta_tag.present? &&
> t.docker_taggable.schema_version == 1) }
> ```

Excuse my ignorance but I run the command as you put it but it returns a syntax error

[root@satellite]# Katello::DockerTag.all.select { (t.schema1_meta_tag.present? && t.docker_taggable.schema_version == 2) || (t.schema2_meta_tag.present? && t.docker_taggable.schema_version == 1) }
bash: syntax error near unexpected token `('

What is the correct way to execute the command?

Regards.

You will need to go inside "foreman-rake console" first and then run the command.

or

# echo "Katello::DockerTag.all.select { (t.schema1_meta_tag.present? && t.docker_taggable.schema_version == 2) || (t.schema2_meta_tag.present? && t.docker_taggable.schema_version == 1) }" | foreman-rake console

Thanks Sayan.

Also, since you're new to the Satellite console, please note that it's an administrative environment where data inconsistencies can be created if the wrong command is run by accident.  The command I sent is a query that causes no modification to your Satellite.

Thanks to both of you.
Without having run foreman-rake katello:correct_docker_meta_tags the result is as follows:

Loading production environment (Rails 6.0.3.4)
Switch to inspect mode.
Katello::DockerTag.all.select { (t.schema1_meta_tag.present? && t.docker_taggable.schema_version == 2) || (t.schema2_meta_tag.present? && t.docker_taggable.schema_version == 1) }
Traceback (most recent call last):
        3: from lib/tasks/console.rake:5:in `block in <top (required)>'
        2: from (irb):1
        1: from (irb):1:in `block in irb_binding'
NameError (undefined local variable or method `t' for main:Object)

With the hotfix applied is it necessary to rerun the satellite-maintain content prepare command?
Because I did not rerun satellite-maintain content prepare after applying the hotfix. 
Directly run satellite-maintain upgrade run --target-version 6.10

Apologies @Caruchamiuller, try this instead:

```
Katello::DockerTag.all.select { |t| (t.schema1_meta_tag.present? && t.docker_taggable.schema_version == 2) || (t.schema2_meta_tag.present? && t.docker_taggable.schema_version == 1) }
```

`satellite-maintain content prepare` shouldn't be required again after applying the hotfix.

And the Satellite services were restarted after applying the hotfix?

Yes, after applying the fix the satellite services were restarted.
Then run `satellite-maintain upgrade upgrade run --target-version 6.10` again.
To fail again with the same error

@iballou 

Does the execution of Katello::DockerTag.all.select ... has any termination output?

Or is it a task that has a delay?

Because it is a few minutes ago without returning anything.

[root@satellite foreman]# echo "Katello::DockerTag.all.select { |t| (t.schema1_meta_tag.present? && t.docker_taggable.schema_version == 2) || (t.schema2_meta_tag.present? && t.docker_taggable.schema_version == 1) }" | foreman-rake console
Loading production environment (Rails 6.0.3.4)
Switch to inspect mode.
Katello::DockerTag.all.select { |t| (t.schema1_meta_tag.present? && t.docker_taggable.schema_version == 2) || (t.schema2_meta_tag.present? && t.docker_taggable.schema_version == 1) }

Hi Caruchamiuller, it is a task with a delay indeed. It scales with the number of docker tags that you have in your environment.  It's a DB query that's run against all of your docker tags.

If it's taking too long, we could have it break at the first corrupt unit with the following modified command:

Katello::DockerTag.all.find { |t| (t.schema1_meta_tag.present? && t.docker_taggable.schema_version == 2) || (t.schema2_meta_tag.present? && t.docker_taggable.schema_version == 1) }

@iballou Thanks

Yes, it took a long time but I ended up with a lot of lines. I understand that these are all the corrupted files that are still on the system.

Does `foreman-rake katello:correct_docker_meta_tags` have any impact on the system still running version 6.9.9?

I am looking into running it to see what output it gives. But first I would like to know if it has any impact.

Regards

@Caruchamiuller,

`foreman-rake katello:correct_docker_meta_tags` makes adjustments to your Docker repositories in the backend that are destructive. Essentially, corrupt units are removed and then recreated. In the meantime while the task is running, your affected repositories are in a temporary unknown state and shouldn't be synced, deleted, published in content views, etc.

The task itself will not give any meaningful output, but it would be helpful to run the command that I gave you to check again for corrupt units.

If you're still seeing corrupt units, we would need to debug further in the Satellite database to determine why the updated code isn't correcting the issue.

@iballou 

Hi,
I ran the command `foreman-rake katello:correct_docker_meta_tags`.

After that, I ran again 
Katello::DockerTag.all.select { |t| (t.schema1_meta_tag.present? && t.docker_taggable.schema_version == 2) || (t.schema2_meta_tag.present? && t.docker_taggable.schema_version == 1) }

The output of the above command returned nothing.

I ran the Satellite upgrade again but it failed again with the same error. The only difference is that I changed the Key ID.

[2022-08-19 19:32:29-0300 #6705] ERROR -- : Failed executing foreman-rake katello:pulp3_content_switchover, exit status 1:
 rake aborted!
ActiveRecord::InvalidForeignKey: PG::ForeignKeyViolation: ERROR:  update or delete on table "katello_docker_meta_tags" violates foreign key constraint "fk_rails_af711a226d" on table "katello_repository_docker_meta_tags"
DETAIL:  Key (id)=(88210) is still referenced from table "katello_repository_docker_meta_tags"

Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/35233 has been resolved.

Automation is going to keep moving it to POST... please ignore it while I continue looking into the latest issue.

hi all, 

I am TAM of Services Australia. Services Australia is our Australian Federal Government customer - biggest customer at APAC. 

Could we please take this as priority? Our customer has wait for this get solved for a long time, stack and shift team also are impacted. They have experienced with corrupted container images for a log time, this will be fixed in 6.10. 

they spent 4 months to upgrade the Satellite servers case stuck the bug for long time, current customer temperature is red. 

Could we please solve this bug as first priority? Thank you. 

Weinan Zhang 
Technical Account Manager - Services Australia 
Red Hat

Created attachment 1910326 [details]
Updated Hotfix RPM for Satellite 6.9.9

INSTALL INSTRUCTIONS:

1. Take a complete backup or snapshot of Satellite 6.9.9 server

2. Download the updated hotfix RPM for Satellite 6.9.9 attached to this BZ and copy it to Satellite server

3. # yum install ./tfm-rubygem-katello-3.18.1.53-3.HOTFIXRHBZ2081560.el7sat.noarch.rpm --disableplugin=foreman-protector

4. # satellite-maintain service restart

AFTER INSTALLATION:

The new logic to correct docker meta tags will run automatically during the switchover to pulp3 content. Users who instead want to perform the cleanup immediately without yet running the full content switchover can do so using the command `foreman-rake katello:correct_docker_meta_tags`

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.9.10 Async Security Update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:8532