Description of problem: Pulpcore migration fails on Satellite install for remotedb with "django.db.utils.OperationalError: could not open extension control file "/usr/share/pgsql/extension/hstore.control": No such file or directory" error. Version-Release number of selected component (if applicable): - Satellite 6.14.0 snap 4 How reproducible: - Always Steps to Reproduce: 1. Try installing Satellite with remotedb Actual results: Applying core.0098_pulp_labels...Traceback (most recent call last): File "/usr/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedFile: could not open extension control file "/usr/share/pgsql/extension/hstore.control": No such file or directory Expected results: - No error - Satellite installation completed successfully. Additional info: # sudo -u pulp PULP_SETTINGS='/etc/pulp/settings.py' pulpcore-manager migrate --noinput Operations to perform: Apply all migrations: ansible, auth, certguard, container, contenttypes, core, file, rpm, sessions Running migrations: Applying core.0098_pulp_labels...Traceback (most recent call last): File "/usr/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedFile: could not open extension control file "/usr/share/pgsql/extension/hstore.control": No such file or directory The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/bin/pulpcore-manager", line 33, in <module> sys.exit(load_entry_point('pulpcore==3.22.2', 'console_scripts', 'pulpcore-manager')()) File "/usr/lib/python3.9/site-packages/pulpcore/app/manage.py", line 11, in manage execute_from_command_line(sys.argv) File "/usr/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line utility.execute() File "/usr/lib/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/usr/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv self.execute(*args, **cmd_options) File "/usr/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute output = self.handle(*args, **options) File "/usr/lib/python3.9/site-packages/django/core/management/base.py", line 89, in wrapped res = handle_func(*args, **kwargs) File "/usr/lib/python3.9/site-packages/django/core/management/commands/migrate.py", line 244, in handle post_migrate_state = executor.migrate( File "/usr/lib/python3.9/site-packages/django/db/migrations/executor.py", line 117, in migrate state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial) File "/usr/lib/python3.9/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial) File "/usr/lib/python3.9/site-packages/django/db/migrations/executor.py", line 227, in apply_migration state = migration.apply(state, schema_editor) File "/usr/lib/python3.9/site-packages/django/db/migrations/migration.py", line 126, in apply operation.database_forwards(self.app_label, schema_editor, old_state, project_state) File "/usr/lib/python3.9/site-packages/django/contrib/postgres/operations.py", line 25, in database_forwards schema_editor.execute( File "/usr/lib/python3.9/site-packages/django/db/backends/base/schema.py", line 145, in execute cursor.execute(sql, params) File "/usr/lib/python3.9/site-packages/django/db/backends/utils.py", line 66, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/usr/lib/python3.9/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers return executor(sql, params, many, context) File "/usr/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/usr/lib/python3.9/site-packages/django/db/utils.py", line 90, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/usr/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.OperationalError: could not open extension control file "/usr/share/pgsql/extension/hstore.control": No such file or directory
what's the output for ` rpm -qa | grep postgres`
# rpm -qa | grep postgres foreman-postgresql-3.7.0-0.5.rc3.el8sat.noarch postgresql-12.12-1.module+el8.6.0+16796+0abe6678.x86_64
please try installing `postgresql-contrib`
# dnf --disableplugin foreman-protector install postgresql-contrib ... Installed: postgresql-contrib-12.12-1.module+el8.6.0+16796+0abe6678.x86_64 uuid-1.6.2-43.el8.x86_64 # sudo -u pulp PULP_SETTINGS='/etc/pulp/settings.py' pulpcore-manager migrate --noinput Operations to perform: Apply all migrations: ansible, auth, certguard, container, contenttypes, core, file, rpm, sessions Running migrations: Applying core.0098_pulp_labels...Traceback (most recent call last): File "/usr/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedFile: could not open extension control file "/usr/share/pgsql/extension/hstore.control": No such file or directory >>> installing postgresql-contrib doesn't help
# rpm -qf /usr/share/pgsql/extension/hstore.control postgresql-contrib-12.12-1.module+el8.6.0+16796+0abe6678.x86_64 # ll /usr/share/pgsql/extension/hstore.control -rw-r--r--. 1 root root 158 Oct 3 2022 /usr/share/pgsql/extension/hstore.control # cat /usr/share/pgsql/extension/hstore.control # hstore extension comment = 'data type for storing sets of (key, value) pairs' default_version = '1.6' module_pathname = '$libdir/hstore' relocatable = true >>> /usr/share/pgsql/extension/hstore.control file is present
Ah, comment#5 and comment#6 reflect the situation on the Satellite, but on the DB Host it is different there: @DB Host: # rpm -qa | grep postgres postgresql-server-12.12-1.module+el8.6.0+16796+0abe6678.x86_64 postgresql-evr-0.0.2-1.el8sat.x86_64 postgresql-12.12-1.module+el8.6.0+16796+0abe6678.x86_64 # dnf --disableplugin foreman-protector install postgresql-contrib ... Installed: libxslt-1.1.32-6.el8.x86_64 postgresql-contrib-12.12-1.module+el8.6.0+16796+0abe6678.x86_64 uuid-1.6.2-43.el8.x86_64 Complete! # ll /usr/share/pgsql/extension/hstore.control -rw-r--r--. 1 root root 158 Oct 3 2022 /usr/share/pgsql/extension/hstore.control @Satellite: # sudo -u pulp PULP_SETTINGS='/etc/pulp/settings.py' pulpcore-manager migrate --noinput Operations to perform: Apply all migrations: ansible, auth, certguard, container, contenttypes, core, file, rpm, sessions Running migrations: Applying core.0098_pulp_labels... OK Applying file.0015_allow_null_manifest... OK Applying rpm.0046_rbac_perms... OK Applying rpm.0047_modulemd_datefield... OK Applying rpm.0048_artifacts_dependencies_fix... OK ... >>> after installing postgresql-contrib on the db host the migration core.0098_pulp_labels runs successfully from the Satellite (-> packaging issue)
Sweet, changing the component to Packaging.
I think this is a documentation issue. Best I can tell, we don't have anything that controls installing packages on an external database server. Upstream katello documentation suggests manually installing postgresql-contrib along with postgresql: https://theforeman.org/plugins/katello/nightly/user_guide/remote_databases/index.html But our official docs for satellite, as of 6.13, do not mention postgresql-contrib in the external database: https://access.redhat.com/documentation/en-us/red_hat_satellite/6.13/html/installing_satellite_server_in_a_connected_network_environment/performing-additional-configuration#using-external-databases_satellite as far as the remotedb test case, updating the template to include installing this package on the remotedb host is warranted.
Link to the updated and published documents: https://dxp-docp-prod.apps.ext-waf.spoke.prod.us-west-2.aws.paas.redhat.com/documentation/en-us/red_hat_satellite/6.14/html-single/installing_satellite_server_in_a_connected_network_environment/index?lb_target=preview#installing-postgresql_satellite (Preview Link)