Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2216745

Summary: Pulpcore migration fails on Satellite install for remotedb with "django.db.utils.OperationalError: could not open extension control file "/usr/share/pgsql/extension/hstore.control": No such file or directory" error.
Product: Red Hat Satellite Reporter: Jameer Pathan <jpathan>
Component: InstallationAssignee: Malhar Jivrajani <mjivraja>
Status: CLOSED CURRENTRELEASE QA Contact: Satellite QE Team <sat-qe-bz-list>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.14.0CC: agadhave, ahumbe, dalley, egolov, ggainey, ipanova, lpramuk, mjivraja, pcreech, rlavi
Target Milestone: 6.14.0Keywords: Documentation, Regression, Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-07-19 13:11:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jameer Pathan 2023-06-22 12:26:30 UTC
Description of problem:
Pulpcore migration fails on Satellite install for remotedb with "django.db.utils.OperationalError: could not open extension control file "/usr/share/pgsql/extension/hstore.control": No such file or directory" error.

Version-Release number of selected component (if applicable):
- Satellite 6.14.0 snap 4

How reproducible:
- Always

Steps to Reproduce:
1. Try installing Satellite with remotedb

Actual results:

Applying core.0098_pulp_labels...Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedFile: could not open extension control file "/usr/share/pgsql/extension/hstore.control": No such file or directory


Expected results:
- No error
- Satellite installation completed successfully.

Additional info:

# sudo -u pulp PULP_SETTINGS='/etc/pulp/settings.py' pulpcore-manager migrate --noinput
Operations to perform:
  Apply all migrations: ansible, auth, certguard, container, contenttypes, core, file, rpm, sessions
Running migrations:
  Applying core.0098_pulp_labels...Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedFile: could not open extension control file "/usr/share/pgsql/extension/hstore.control": No such file or directory


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/bin/pulpcore-manager", line 33, in <module>
    sys.exit(load_entry_point('pulpcore==3.22.2', 'console_scripts', 'pulpcore-manager')())
  File "/usr/lib/python3.9/site-packages/pulpcore/app/manage.py", line 11, in manage
    execute_from_command_line(sys.argv)
  File "/usr/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/usr/lib/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/usr/lib/python3.9/site-packages/django/core/management/base.py", line 89, in wrapped
    res = handle_func(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/django/core/management/commands/migrate.py", line 244, in handle
    post_migrate_state = executor.migrate(
  File "/usr/lib/python3.9/site-packages/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/usr/lib/python3.9/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/lib/python3.9/site-packages/django/db/migrations/executor.py", line 227, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/usr/lib/python3.9/site-packages/django/db/migrations/migration.py", line 126, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/usr/lib/python3.9/site-packages/django/contrib/postgres/operations.py", line 25, in database_forwards
    schema_editor.execute(
  File "/usr/lib/python3.9/site-packages/django/db/backends/base/schema.py", line 145, in execute
    cursor.execute(sql, params)
  File "/usr/lib/python3.9/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/usr/lib/python3.9/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/lib/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.OperationalError: could not open extension control file "/usr/share/pgsql/extension/hstore.control": No such file or directory

Comment 2 Ina Panova 2023-06-22 15:52:04 UTC
what's the output for ` rpm -qa | grep postgres`

Comment 3 Jameer Pathan 2023-06-23 09:19:38 UTC
# rpm -qa | grep postgres
foreman-postgresql-3.7.0-0.5.rc3.el8sat.noarch
postgresql-12.12-1.module+el8.6.0+16796+0abe6678.x86_64

Comment 4 Ina Panova 2023-06-26 11:48:03 UTC
please try installing `postgresql-contrib`

Comment 5 Lukas Pramuk 2023-06-26 12:55:27 UTC
# dnf --disableplugin foreman-protector install postgresql-contrib
...
Installed:
  postgresql-contrib-12.12-1.module+el8.6.0+16796+0abe6678.x86_64           uuid-1.6.2-43.el8.x86_64               

# sudo -u pulp PULP_SETTINGS='/etc/pulp/settings.py' pulpcore-manager migrate --noinput
Operations to perform:
  Apply all migrations: ansible, auth, certguard, container, contenttypes, core, file, rpm, sessions
Running migrations:
  Applying core.0098_pulp_labels...Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedFile: could not open extension control file "/usr/share/pgsql/extension/hstore.control": No such file or directory

>>> installing postgresql-contrib doesn't help

Comment 6 Lukas Pramuk 2023-06-26 13:06:25 UTC
# rpm -qf /usr/share/pgsql/extension/hstore.control
postgresql-contrib-12.12-1.module+el8.6.0+16796+0abe6678.x86_64

# ll /usr/share/pgsql/extension/hstore.control
-rw-r--r--. 1 root root 158 Oct  3  2022 /usr/share/pgsql/extension/hstore.control

# cat /usr/share/pgsql/extension/hstore.control
# hstore extension
comment = 'data type for storing sets of (key, value) pairs'
default_version = '1.6'
module_pathname = '$libdir/hstore'
relocatable = true

>>> /usr/share/pgsql/extension/hstore.control file is present

Comment 7 Lukas Pramuk 2023-06-27 07:19:40 UTC
Ah, comment#5 and comment#6 reflect the situation on the Satellite, but on the DB Host it is different there:

@DB Host:
# rpm -qa | grep postgres
postgresql-server-12.12-1.module+el8.6.0+16796+0abe6678.x86_64
postgresql-evr-0.0.2-1.el8sat.x86_64
postgresql-12.12-1.module+el8.6.0+16796+0abe6678.x86_64

# dnf --disableplugin foreman-protector install postgresql-contrib
...
Installed:
  libxslt-1.1.32-6.el8.x86_64          postgresql-contrib-12.12-1.module+el8.6.0+16796+0abe6678.x86_64         uuid-1.6.2-43.el8.x86_64                                       

Complete!

# ll /usr/share/pgsql/extension/hstore.control
-rw-r--r--. 1 root root 158 Oct  3  2022 /usr/share/pgsql/extension/hstore.control

@Satellite:
# sudo -u pulp PULP_SETTINGS='/etc/pulp/settings.py' pulpcore-manager migrate --noinput
Operations to perform:
  Apply all migrations: ansible, auth, certguard, container, contenttypes, core, file, rpm, sessions
Running migrations:
  Applying core.0098_pulp_labels... OK
  Applying file.0015_allow_null_manifest... OK
  Applying rpm.0046_rbac_perms... OK
  Applying rpm.0047_modulemd_datefield... OK
  Applying rpm.0048_artifacts_dependencies_fix... OK
...

>>> after installing postgresql-contrib on the db host the migration core.0098_pulp_labels runs successfully from the Satellite (-> packaging issue)

Comment 8 Ina Panova 2023-06-27 08:43:36 UTC
Sweet, changing the component to Packaging.

Comment 9 Patrick Creech 2023-06-27 15:39:25 UTC
I think this is a documentation issue.  Best I can tell, we don't have anything that controls installing packages on an external database server.  Upstream katello documentation suggests manually installing postgresql-contrib along with postgresql:

https://theforeman.org/plugins/katello/nightly/user_guide/remote_databases/index.html

But our official docs for satellite, as of 6.13, do not mention postgresql-contrib in the external database:

https://access.redhat.com/documentation/en-us/red_hat_satellite/6.13/html/installing_satellite_server_in_a_connected_network_environment/performing-additional-configuration#using-external-databases_satellite

as far as the remotedb test case, updating the template to include installing this package on the remotedb host is warranted.