Bug 2161929

Summary:	Locale change caused by RHEL upgrade results in database index corruption "get() returned more than one Modulemd -- it returned 2!"
Product:	Red Hat Satellite	Reporter:	Evgeni Golov <egolov>
Component:	Satellite Maintain	Assignee:	Evgeni Golov <egolov>
Status:	CLOSED ERRATA	QA Contact:	Lukas Pramuk <lpramuk>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	6.11.3	CC:	ahumbe, ajambhul, bdm, dalley, egolov, ehelms, ggainey, hyu, iballou, jbhatia, jentrena, juwatts, lpramuk, mjia, osousa, paji, pdudley, pdwyer, pmoravec, sajha, satellite6-bugs, saydas, wpinheir
Target Milestone:	6.11.5	Keywords:	Triaged, Upgrades, WorkAround
Target Release:	Unused
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	rubygem-foreman_maintain-1.0.19-1	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	2142270
Clones:	2166744 (view as bug list)		Environment:
Last Closed:	2023-03-07 19:00:52 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	2142270
Bug Blocks:

Description Evgeni Golov 2023-01-18 10:59:08 UTC

This is a clone for Foreman Maintain, as that needs to perform the reindexing in the case of a DB restore of a EL7 backup on EL8!


+++ This bug was initially created as a clone of Bug #2142270 +++

Description of problem:
----------------------

Repository synchronization on Red Hat Satellite 6.11.3 on RHEL-8 fails with the following error :

Katello::Errors::Pulp3Error "get() returned more than one Modulemd -- it returned 2!"




Error:
Katello::Errors::Pulp3Error
get() returned more than one Modulemd -- it returned 2!


It seems to be related to the locales change in glibc-2.28 causing the collation changed in Postgresql
--------------
https://dba.stackexchange.com/questions/268027/postgresql-duplicates-despite-having-a-unique-constraint
https://wiki.postgresql.org/wiki/Locale_data_changes
https://postgresql.verite.pro/blog/2018/08/27/glibc-upgrade.html
-------------


I checked 3 cases attached to this bugzilla. They are running Satellite on RHEL 8 which having glibc-2.28.

Looking at the pulp_created column in core_content, the duplicated records were also created after the release date of Satellite 6.11.

It seems like we will need to reindex all the tables to avoid the issue after upgrading the OS to RHEL 8.

Comment 1 Evgeni Golov 2023-01-18 11:00:19 UTC

Created redmine issue https://projects.theforeman.org/issues/35966 from this bug

Comment 2 Odilon Sousa 2023-02-08 15:00:36 UTC

*** Bug 2166744 has been marked as a duplicate of this bug. ***

Comment 4 Lukas Pramuk 2023-02-15 10:36:24 UTC

VERIFIED.

@Satellite 6.11.5 Snap 1
rubygem-foreman_maintain-1.0.19-1.el8sat.noarch

by the following manual reproducer:

1) Have el7 DB backup (6.11.z version)

2) Restore the backup to el8 machine using satellite-clone

# satellite-clone -y
...
 
PLAY RECAP *********************************************************************************************************
localhost                  : ok=75   changed=27   unreachable=0    failed=0    skipped=29   rescued=0    ignored=1  

3) Run DB re-index on el8 Satellite to detect DB corruption

FIX:
# runuser - postgres -c "reindexdb -a"
reindexdb: reindexing database "candlepin"
reindexdb: reindexing database "foreman"
reindexdb: reindexing database "postgres"
reindexdb: reindexing database "pulpcore"
reindexdb: reindexing database "template1"

vs.

REPRO:
# runuser - postgres -c "reindexdb -a"
reindexdb: reindexing database "candlepin"
reindexdb: reindexing database "foreman"
reindexdb: error: reindexing of database "foreman" failed: ERROR:  could not create unique index "index_fact_names_on_name_and_type"
DETAIL:  Key (name, type)=(sshrsakey, PuppetFactName) is duplicated.


>>> satellite-clone/satellite-maintain restores DB backup on RHEL8 w/o indices corruption

Comment 9 errata-xmlrpc 2023-03-07 19:00:52 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Critical: Satellite 6.11.5 Async Security Update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:1151

Comment 10 Ian Ballou 2023-04-10 21:00:44 UTC

*** Bug 2172655 has been marked as a duplicate of this bug. ***

Comment 11 Daniel Alley 2024-01-12 14:56:20 UTC

*** Bug 2247864 has been marked as a duplicate of this bug. ***