Description of problem: During 6.7 > 6.8 upgrade, foreman-rake db:migrate run 20200402130013_add_repsoitory_docker_meta_tag_f_key which comes from katello. The migrations script calls: Katello::RepositoryDockerTag.where.not(:docker_tag_id => Katello::DockerTag.select(:id)).delete_all and it was hanging forever. This is because there are 2 million+ records in the katello_docker_tags table: foreman=# select count(*) from katello_docker_tags; count --------- 2315209 (1 row) As a consequence, the upgrade failed with timeout error and hit this bug: https://bugzilla.redhat.com/show_bug.cgi?id=1904963 Version-Release number of selected component (if applicable): 6.7 How reproducible: on the customer's db Steps to Reproduce: On a reproducer with the customers' db, foreman-rake console` Katello::RepositoryDockerTag.where.not(:docker_tag_id => Katello::DockerTag.select(:id)) This is hanging forever and never complete. At the same time, postgres is consuming 100% cpu. Actual results: Upgrade fails on the db migration Expected results: Upgrade should complete.
Connecting redmine issue https://projects.theforeman.org/issues/33012 from this bug
Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/33012 has been resolved.
Development wasn't able to reproduce this issue to the level that was described by the customer using dummy data, however we were able to improve the performance considerably from the small amount of time we did see with a large amount of data.
This BZ is not relevant for 6.10.0 and thus cannot be verified. BZ delivers fix in the following migrations file: 20200402130013_add_repsoitory_docker_meta_tag_f_key.rb This migration is already present in DB since 6.8.0 while on 6.7.5 it's not there yet Therefore the fix is usefull only for 6.8.z+ (performed during 6.7 -> 6.8 upgrade) For 6.10.0 we just deliver the same file (*.rb) but with no effect, it never can be re-executed again
Closing based upon comment 7.