Bug 2216745

Summary: Pulpcore migration fails on Satellite install for remotedb with "django.db.utils.OperationalError: could not open extension control file "/usr/share/pgsql/extension/hstore.control": No such file or directory" error.
Product: Red Hat Satellite Reporter: Jameer Pathan <jpathan>
Component: InstallerAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED CURRENTRELEASE QA Contact: Satellite QE Team <sat-qe-bz-list>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.14.0CC: agadhave, ahumbe, dalley, egolov, ggainey, ipanova, lpramuk, mjivraja, pcreech, rlavi
Target Milestone: 6.14.0Keywords: Documentation, Regression, Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-07-19 13:11:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jameer Pathan 2023-06-22 12:26:30 UTC
Description of problem:
Pulpcore migration fails on Satellite install for remotedb with "django.db.utils.OperationalError: could not open extension control file "/usr/share/pgsql/extension/hstore.control": No such file or directory" error.

Version-Release number of selected component (if applicable):
- Satellite 6.14.0 snap 4

How reproducible:
- Always

Steps to Reproduce:
1. Try installing Satellite with remotedb

Actual results:

Applying core.0098_pulp_labels...Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedFile: could not open extension control file "/usr/share/pgsql/extension/hstore.control": No such file or directory


Expected results:
- No error
- Satellite installation completed successfully.

Additional info:

# sudo -u pulp PULP_SETTINGS='/etc/pulp/settings.py' pulpcore-manager migrate --noinput
Operations to perform:
  Apply all migrations: ansible, auth, certguard, container, contenttypes, core, file, rpm, sessions
Running migrations:
  Applying core.0098_pulp_labels...Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedFile: could not open extension control file "/usr/share/pgsql/extension/hstore.control": No such file or directory


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/bin/pulpcore-manager", line 33, in <module>
    sys.exit(load_entry_point('pulpcore==3.22.2', 'console_scripts', 'pulpcore-manager')())
  File "/usr/lib/python3.9/site-packages/pulpcore/app/manage.py", line 11, in manage
    execute_from_command_line(sys.argv)
  File "/usr/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/usr/lib/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/usr/lib/python3.9/site-packages/django/core/management/base.py", line 89, in wrapped
    res = handle_func(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/django/core/management/commands/migrate.py", line 244, in handle
    post_migrate_state = executor.migrate(
  File "/usr/lib/python3.9/site-packages/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/usr/lib/python3.9/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/lib/python3.9/site-packages/django/db/migrations/executor.py", line 227, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/usr/lib/python3.9/site-packages/django/db/migrations/migration.py", line 126, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/usr/lib/python3.9/site-packages/django/contrib/postgres/operations.py", line 25, in database_forwards
    schema_editor.execute(
  File "/usr/lib/python3.9/site-packages/django/db/backends/base/schema.py", line 145, in execute
    cursor.execute(sql, params)
  File "/usr/lib/python3.9/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/usr/lib/python3.9/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/lib/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.OperationalError: could not open extension control file "/usr/share/pgsql/extension/hstore.control": No such file or directory

Comment 2 Ina Panova 2023-06-22 15:52:04 UTC
what's the output for ` rpm -qa | grep postgres`

Comment 3 Jameer Pathan 2023-06-23 09:19:38 UTC
# rpm -qa | grep postgres
foreman-postgresql-3.7.0-0.5.rc3.el8sat.noarch
postgresql-12.12-1.module+el8.6.0+16796+0abe6678.x86_64

Comment 4 Ina Panova 2023-06-26 11:48:03 UTC
please try installing `postgresql-contrib`

Comment 5 Lukas Pramuk 2023-06-26 12:55:27 UTC
# dnf --disableplugin foreman-protector install postgresql-contrib
...
Installed:
  postgresql-contrib-12.12-1.module+el8.6.0+16796+0abe6678.x86_64           uuid-1.6.2-43.el8.x86_64               

# sudo -u pulp PULP_SETTINGS='/etc/pulp/settings.py' pulpcore-manager migrate --noinput
Operations to perform:
  Apply all migrations: ansible, auth, certguard, container, contenttypes, core, file, rpm, sessions
Running migrations:
  Applying core.0098_pulp_labels...Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedFile: could not open extension control file "/usr/share/pgsql/extension/hstore.control": No such file or directory

>>> installing postgresql-contrib doesn't help

Comment 6 Lukas Pramuk 2023-06-26 13:06:25 UTC
# rpm -qf /usr/share/pgsql/extension/hstore.control
postgresql-contrib-12.12-1.module+el8.6.0+16796+0abe6678.x86_64

# ll /usr/share/pgsql/extension/hstore.control
-rw-r--r--. 1 root root 158 Oct  3  2022 /usr/share/pgsql/extension/hstore.control

# cat /usr/share/pgsql/extension/hstore.control
# hstore extension
comment = 'data type for storing sets of (key, value) pairs'
default_version = '1.6'
module_pathname = '$libdir/hstore'
relocatable = true

>>> /usr/share/pgsql/extension/hstore.control file is present

Comment 7 Lukas Pramuk 2023-06-27 07:19:40 UTC
Ah, comment#5 and comment#6 reflect the situation on the Satellite, but on the DB Host it is different there:

@DB Host:
# rpm -qa | grep postgres
postgresql-server-12.12-1.module+el8.6.0+16796+0abe6678.x86_64
postgresql-evr-0.0.2-1.el8sat.x86_64
postgresql-12.12-1.module+el8.6.0+16796+0abe6678.x86_64

# dnf --disableplugin foreman-protector install postgresql-contrib
...
Installed:
  libxslt-1.1.32-6.el8.x86_64          postgresql-contrib-12.12-1.module+el8.6.0+16796+0abe6678.x86_64         uuid-1.6.2-43.el8.x86_64                                       

Complete!

# ll /usr/share/pgsql/extension/hstore.control
-rw-r--r--. 1 root root 158 Oct  3  2022 /usr/share/pgsql/extension/hstore.control

@Satellite:
# sudo -u pulp PULP_SETTINGS='/etc/pulp/settings.py' pulpcore-manager migrate --noinput
Operations to perform:
  Apply all migrations: ansible, auth, certguard, container, contenttypes, core, file, rpm, sessions
Running migrations:
  Applying core.0098_pulp_labels... OK
  Applying file.0015_allow_null_manifest... OK
  Applying rpm.0046_rbac_perms... OK
  Applying rpm.0047_modulemd_datefield... OK
  Applying rpm.0048_artifacts_dependencies_fix... OK
...

>>> after installing postgresql-contrib on the db host the migration core.0098_pulp_labels runs successfully from the Satellite (-> packaging issue)

Comment 8 Ina Panova 2023-06-27 08:43:36 UTC
Sweet, changing the component to Packaging.

Comment 9 Patrick Creech 2023-06-27 15:39:25 UTC
I think this is a documentation issue.  Best I can tell, we don't have anything that controls installing packages on an external database server.  Upstream katello documentation suggests manually installing postgresql-contrib along with postgresql:

https://theforeman.org/plugins/katello/nightly/user_guide/remote_databases/index.html

But our official docs for satellite, as of 6.13, do not mention postgresql-contrib in the external database:

https://access.redhat.com/documentation/en-us/red_hat_satellite/6.13/html/installing_satellite_server_in_a_connected_network_environment/performing-additional-configuration#using-external-databases_satellite

as far as the remotedb test case, updating the template to include installing this package on the remotedb host is warranted.