Bug 1828931 - engine-vacuum fails with permission denied for schema pg_temp_23
Summary: engine-vacuum fails with permission denied for schema pg_temp_23
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-engine
Classification: oVirt
Component: General
Version: 4.4.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ovirt-4.4.1
: 4.4.1
Assignee: Eli Mesika
QA Contact: Petr Matyáš
URL:
Whiteboard:
Depends On:
Blocks: 1837207
TreeView+ depends on / blocked
 
Reported: 2020-04-28 15:31 UTC by Petr Matyáš
Modified: 2020-07-08 08:25 UTC (History)
3 users (show)

Fixed In Version: ovirt-engine-4.4.1
Doc Type: No Doc Update
Doc Text:
Clone Of:
: 1837207 (view as bug list)
Environment:
Last Closed: 2020-07-08 08:25:04 UTC
oVirt Team: Infra
Embargoed:
pm-rhel: ovirt-4.4+
pm-rhel: blocker?


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 108856 0 master MERGED db: cleaning orphan temp schemas 2020-08-19 12:53:23 UTC
oVirt gerrit 109010 0 master MERGED db: drop temp table ASAP 2020-08-19 12:53:23 UTC

Description Petr Matyáš 2020-04-28 15:31:42 UTC
Description of problem:
When running engine-vacuum command it fails with permission denied for schema pg_temp_23 and various other schemas, I reproduced this with pg_temp_19, pg_temp_9 and pg_temp_23.

Version-Release number of selected component (if applicable):
ovirt-engine-tools-4.4.0-0.33.master.el8ev.noarch

How reproducible:
always

Steps to Reproduce:
1. run engine-vacuum
2.
3.

Actual results:
vacuumdb: vacuuming database "engine"
vacuumdb: error: vacuuming of table "pg_temp_23.tt_vm_pool_group" in database "engine" failed: ERROR:  permission denied for schema pg_temp_23

Expected results:
should vacuum the DB without errors

Additional info:

Comment 1 Eli Mesika 2020-05-04 14:47:03 UTC
engine-vacuum actually calls PG's vacuumdb utility 
I checked running directly vacuumdb with user 'engine' on the 'engine' db 

/usr/bin/vacuumdb -d engine -U engine

vacuumdb: error: vacuuming of table "pg_temp_6.tt_vm_pool_running" in database "engine" failed: ERROR:  permission denied for schema pg_temp_6

googling this a bit I found that [1]

So, this seems as a PG regression that forces vacuumdb to use postgres admin user to perform vacuum 

[1] https://github.com/greenplum-db/gpdb/pull/9304

Comment 2 Eli Mesika 2020-05-04 17:31:11 UTC
It seems like orphan schemas are left in the database , those schemas are not owned by the 'engine' role and can not be dropped by it 

engine=> select nspname from pg_namespace where nspname like 'pg_temp%';
  nspname   
------------
 pg_temp_1
 pg_temp_11
 pg_temp_12
 pg_temp_8
 pg_temp_6
 pg_temp_3
 pg_temp_14
 pg_temp_13
 pg_temp_4
 pg_temp_7
(10 rows)

The drop statements for those orphan schemas can be extracted to a sql script by;

 psql -U engine -Atc "select 'drop schema if exists ' || nspname || ' cascade;' from (select nspname from pg_namespace where nspname like 'pg_temp%') as foo" engine > drop_orphan_schemas.sql

Then we can run psql to execute this script before running vacuumdb , but still , the sql script must run from psql with the 'postgres' user

Comment 3 RHEL Program Management 2020-05-06 12:50:20 UTC
This bug report has Keywords: Regression or TestBlocker.
Since no regressions or test blockers are allowed between releases, it is also being identified as a blocker for this release. Please resolve ASAP.

Comment 4 RHEL Program Management 2020-05-11 11:14:14 UTC
The documentation text flag should only be set after 'doc text' field is provided. Please provide the documentation text and set the flag to '?' again.

Comment 5 Eli Mesika 2020-05-11 11:18:40 UTC
This paragraph should be added to the release doc:


For remote vacuuming if you got errors like 'permission denied for schema pg_temp_XX' please do the following:
     1) log in into the remote database machine
     2) run
           psql  -U <db-admin-role> -Atc \"select 'drop schema if exists ' || nspname || ' cascade;'
           from (select distinct nspname from pg_class join pg_namespace on (relnamespace=pg_namespace.oid)
           where pg_is_other_temp_schema(relnamespace)) as foo\" <engine database name > <temporary file>
     3) run
           psql  -U <db-admin-role> -f <temporary file>
     4) try to run engine-vacuum again"

Comment 6 Martin Perina 2020-05-11 13:20:20 UTC
(In reply to Eli Mesika from comment #5)
> This paragraph should be added to the release doc:
> 
> 
> For remote vacuuming if you got errors like 'permission denied for schema
> pg_temp_XX' please do the following:
>      1) log in into the remote database machine
>      2) run
>            psql  -U <db-admin-role> -Atc \"select 'drop schema if exists '
> || nspname || ' cascade;'
>            from (select distinct nspname from pg_class join pg_namespace on
> (relnamespace=pg_namespace.oid)
>            where pg_is_other_temp_schema(relnamespace)) as foo\" <engine
> database name > <temporary file>
>      3) run
>            psql  -U <db-admin-role> -f <temporary file>
>      4) try to run engine-vacuum again"

Steven, we should also add this as a step into 4.4 upgrade guide to upgrade remote database chapters. Probably after restoring 4.3 database backup and upgrading this database from 10.6 to 12 and before running engine-setup.

Comment 7 Steve Goodman 2020-05-11 14:06:11 UTC
OK. I added this comment to bug 1802650.

Comment 9 Petr Matyáš 2020-05-22 12:37:44 UTC
Verified on ovirt-engine-4.4.1-0.1.el8ev.noarch

Comment 10 Sandro Bonazzola 2020-07-08 08:25:04 UTC
This bugzilla is included in oVirt 4.4.1 release, published on July 8th 2020.

Since the problem described in this bug report should be resolved in oVirt 4.4.1 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.