Bug 1936775

Summary: 6.7 > 6.8 upgrade fails when running the db migration(20200402130013_add_repsoitory_docker_meta_tag_f_key)
Product: Red Hat Satellite Reporter: matt jia <mjia>
Component: RepositoriesAssignee: James Jeffers <jjeffers>
Status: CLOSED CURRENTRELEASE QA Contact: Lukas Pramuk <lpramuk>
Severity: high Docs Contact:
Priority: high    
Version: 6.7.0CC: ahumbe, jjeffers, jsherril, juwatts, lvrtelov, pcreech
Target Milestone: 6.10.0Keywords: Triaged, Upgrades
Target Release: Unused   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-11-01 10:14:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description matt jia 2021-03-09 06:41:00 UTC 
Description of problem:

During 6.7 > 6.8 upgrade, foreman-rake db:migrate  run 20200402130013_add_repsoitory_docker_meta_tag_f_key which comes from katello.
The migrations script calls:

Katello::RepositoryDockerTag.where.not(:docker_tag_id => Katello::DockerTag.select(:id)).delete_all

and it was hanging forever.

This is because there are 2 million+ records in the katello_docker_tags table:

foreman=# select count(*) from katello_docker_tags;
  count  
---------
 2315209
(1 row)

As a consequence, the upgrade failed with timeout error and hit this bug:

https://bugzilla.redhat.com/show_bug.cgi?id=1904963

Version-Release number of selected component (if applicable):

6.7

How reproducible:

on the customer's db

Steps to Reproduce:

On a reproducer with the customers' db,  

foreman-rake console`
Katello::RepositoryDockerTag.where.not(:docker_tag_id => Katello::DockerTag.select(:id)) 

This is hanging forever and never complete. At the same time, postgres is consuming 100% cpu.

Actual results:

Upgrade fails on the db migration

Expected results:

Upgrade should complete.
Comment 4 James Jeffers 2021-07-13 20:31:23 UTC 
Connecting redmine issue https://projects.theforeman.org/issues/33012 from this bug
Comment 5 Bryan Kearney 2021-08-02 16:04:40 UTC 
Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/33012 has been resolved.
Comment 6 Justin Sherrill 2021-10-26 15:43:26 UTC 
Development wasn't able to reproduce this issue to the level that was described by the customer using dummy data, however we were able to improve the performance considerably from the small amount of time we did see with a large amount of data.
Comment 7 Lukas Pramuk 2021-10-27 08:01:13 UTC 
This BZ is not relevant for 6.10.0 and thus cannot be verified.

BZ delivers fix in the following migrations file:
  20200402130013_add_repsoitory_docker_meta_tag_f_key.rb 

This migration is already present in DB since 6.8.0 while on 6.7.5 it's not there yet
Therefore the fix is usefull only for 6.8.z+ (performed during 6.7 -> 6.8 upgrade)

For 6.10.0 we just deliver the same file (*.rb) but with no effect, it never can be re-executed again
Comment 8 Lucie Vrtelova 2021-11-01 10:14:28 UTC 
Closing based upon comment 7.