Bug 2188239 - Needs more clarification about "You can move the PostgreSQL data on your own " in case of Leapp + Satellite 6.11 [NEEDINFO]
Summary: Needs more clarification about "You can move the PostgreSQL data on your own ...
Keywords:
Status: NEW
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Upgrades
Version: 6.11.0
Hardware: All
OS: All
high
medium
Target Milestone: Unspecified
Assignee: Evgeni Golov
QA Contact: Satellite QE Team
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-04-20 08:23 UTC by Direct Docs Feedback
Modified: 2023-08-10 12:42 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:
Embargoed:
mdolezel: needinfo? (agadhave)
mdolezel: needinfo? (mjivraja)


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github oamg leapp-repository pull 1071 0 None Draft BZ#2188239 - better detect already migrated PostgreSQL data 2023-07-20 05:36:22 UTC
Red Hat Issue Tracker SAT-17243 0 None None None 2023-04-20 08:23:45 UTC
Red Hat Issue Tracker SAT-17264 0 None None None 2023-08-10 12:42:34 UTC

Description Direct Docs Feedback 2023-04-20 08:23:25 UTC
Needs more clarification about this statement w.r.t leapp and active services of satellite prior executing leapp

Reported by: rhn-support-saydas

https://access.redhat.com/documentation/en-us/red_hat_satellite/6.11/html/upgrading_and_updating_red_hat_satellite/upgrading-satellite-or-proxy-in-place-using-leapp_upgrade-guide#annotations:e67ed298-3aa1-46bc-87a2-e95fde05bed1

Comment 1 Sayan Das 2023-04-20 08:30:18 UTC

The last point of the prerequisites is 

~~
During the upgrade, the PostgreSQL data is moved from /var/opt/rh/rh-postgresql12/lib/pgsql/data/ to /var/lib/pgsql/data/. If these two paths reside on the same partition, no further action is required. If they reside on different partitions, ensure that there is enough space for the data to be copied over. You can move the PostgreSQL data on your own and the upgrade will skip this step if /var/opt/rh/rh-postgresql12/lib/pgsql/data/ does not exist.
~~

What is exactly meant by this and what precautions users would need to take ?

~~
You can move the PostgreSQL data on your own and the upgrade will skip this step if /var/opt/rh/rh-postgresql12/lib/pgsql/data/ does not exist.
~~

If a user take this statement on its face value, then for the running 6.11 satellite on RHEL 7,

* They will move ( using mv or rsync ) /var/opt/rh/rh-postgresql12/lib/pgsql/data/ content to /var/lib/pgsql/data/ ( which is in a new FS )

* Delete /var/opt/rh/rh-postgresql12/lib/pgsql/data/

* That will cause the running satellite to completely break down i.e. 

~~
Displaying the following service(s):
rh-redis5-redis, postgresql, pulpcore-api, pulpcore-content, pulpcore-worker, pulpcore-worker, pulpcore-worker, pulpcore-worker, pulpcore-worker, pulpcore-worker, pulpcore-worker, pulpcore-worker, tomcat, dynflow-sidekiq@orchestrator, foreman, httpd, puppetserver, dynflow-sidekiq@worker-1, dynflow-sidekiq@worker-hosts-queue-1, foreman-proxy
| displaying rh-redis5-redis                       [OK]
/ displaying postgresql                            [FAIL]
/ displaying pulpcore-api                          [OK]
/ displaying pulpcore-content                      [OK]


    displaying pulpcore-worker             [FAIL]

    displaying pulpcore-worker             [FAIL]

    displaying pulpcore-worker             [FAIL]

    displaying pulpcore-worker             [FAIL]

    displaying pulpcore-worker             [FAIL]

    displaying pulpcore-worker             [FAIL]

    displaying pulpcore-worker             [FAIL]

    displaying pulpcore-worker             [FAIL]

    displaying tomcat                                [OK]
    \ displaying dynflow-sidekiq@orchestrator          [OK]
    \ displaying foreman                               [OK]
    \ displaying httpd                                 [OK]
    \ displaying puppetserver                          [OK]
    \ displaying dynflow-sidekiq@worker-1              [OK]
    \ displaying dynflow-sidekiq@worker-hosts-queue-1  [OK]
    | displaying foreman-proxy                         [OK]
    | All services displayed                                              [FAIL]
    Some services are not running (postgresql, pulpcore-worker, pulpcore-worker, pulpcore-worker, pulpcore-worker, pulpcore-worker, pulpcore-worker, pulpcore-worker, pulpcore-worker)


~~


* And then if the user tries leapp preupgrade, It will say 

~~
Process Process-218:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python2.7/site-packages/leapp/repository/actor_definition.py", line 72, in _do_run
    actor_instance.run(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/leapp/actors/init.py", line 289, in run
    self.process(*args)
  File "/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/actors/satellite_upgrade_facts/actor.py", line 96, in process
    scl_psql_stat = os.stat(POSTGRESQL_SCL_DATA_PATH)
OSError: [Errno 2] No such file or directory: '/var/opt/rh/rh-postgresql12/lib/pgsql/data/'

========================================================================================================
Actor satellite_upgrade_facts unexpectedly terminated with exit code: 1 - Please check the above details
========================================================================================================
~~


So either what we are saying is not correct i.e. 
~~
You can move the PostgreSQL data on your own and the upgrade will skip this step if /var/opt/rh/rh-postgresql12/lib/pgsql/data/ does not exist.
~~

Or, there a lot of explanation we need to put to support our statement there.

Comment 2 Marie Hornickova 2023-04-20 15:19:22 UTC
Hello,
Many thanks for using the direct documentation feedback tool to provide this suggestion for improvement. 
Updates on the progress on the implementation will be shared once the team starts working on it.
Thank you!

Comment 4 Evgeni Golov 2023-04-21 08:37:18 UTC
Heh, so I think what you found is a bug in our leapp actor, not the docs!

The idea is really:
* foreman-maintain services stop
* <do whatever is needed to move over the data from /var/opt/rh/rh-postgresql12/lib/pgsql/data/ to /var/lib/pgsql/data/> (this can be a remount of the dedicated storage, rsync, whatever, the admin has to know their deployment, not us)
* leapp preupgrade
* leapp upgrade

The way I tested this (and why it worked for a lot of people):
* /var/opt/rh/rh-postgresql12/lib/pgsql/data/ is a dedicated mountpoint
* one creates /var/lib/pgsql/data/
* umount /var/opt/rh/rh-postgresql12/lib/pgsql/data/
* mount /var/lib/pgsql/data/

This leaves /var/opt/rh/rh-postgresql12/lib/pgsql/data as an existing, empty, folder. So all the code still works (as it *wrongly* expects it to always exist) and the upgrade succeeds.

However, you are absolutely right, if the admin manually moved the data, there is no reason /var/opt/rh/rh-postgresql12/lib/pgsql/data would still exist.

Sayan, would you say if that part is a bug, the rest of the docs are clear enough? Or should we also adjust these further?

Comment 5 Sayan Das 2023-04-21 09:07:00 UTC
So if i am understanding correctly, The idea is to plan the activity in one shot i.e. 

* Take a MW

* Stop all services 

* MOve all stuff from /var/opt/rh/rh-postgresql12/lib/pgsql/data/ ( old FS ) to /var/lib/pgsql/data/  ( new FS )

* Ensure /var/opt/rh/rh-postgresql12/lib/pgsql/data/ directory itself is not deleted but only emptied out 

* leapp preupgrade

* fix inhibitors 

* leapp upgrade + reboot 

* Ensure the installer completes as a part of leapp_resume 

* post-upgrade steps. 



Now, Here's where the issue is 

We say
~~
You can move the PostgreSQL data on your own and the upgrade will skip this step if /var/opt/rh/rh-postgresql12/lib/pgsql/data/ does not exist.
~~

But we don't clearly mention\highlight the fact that leapp won't work without the directory /var/opt/rh/rh-postgresql12/lib/pgsql/data/ itself being present .

So someone can move the data and then simply unount the /var/opt/rh/rh-postgresql12/lib/pgsql/data/ or delete it , and then run leapp which will fail with 

  File "/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/actors/satellite_upgrade_facts/actor.py", line 96, in process
    scl_psql_stat = os.stat(POSTGRESQL_SCL_DATA_PATH)
OSError: [Errno 2] No such file or directory: '/var/opt/rh/rh-postgresql12/lib/pgsql/data/'



So if we only consider this as a DOC BUG where we are missing to provide the required specifics of the manual migration, Then Yes, Even a clarification in the docs is good enough for me. 

In that way, we may not need to deal with leapp itself but only take care of following the right procedure. 

Even if we consider the behavior of leapp as a bug, The first priority is to fix the documentation ( as that is referred by users almost everyday ) and then we can work on a BZ related to leapp itself.

Comment 6 Evgeni Golov 2023-04-21 11:56:18 UTC
(In reply to Sayan Das from comment #5)
> So if i am understanding correctly, The idea is to plan the activity in one
> shot i.e. 
> 
> * Take a MW

MW?

> 
> * Stop all services 
> 
> * MOve all stuff from /var/opt/rh/rh-postgresql12/lib/pgsql/data/ ( old FS )
> to /var/lib/pgsql/data/  ( new FS )
> 
> * Ensure /var/opt/rh/rh-postgresql12/lib/pgsql/data/ directory itself is not
> deleted but only emptied out 
> 
> * leapp preupgrade
> 
> * fix inhibitors 
> 
> * leapp upgrade + reboot 
> 
> * Ensure the installer completes as a part of leapp_resume 
> 
> * post-upgrade steps. 

Correct.


> Now, Here's where the issue is 
> 
> We say
> ~~
> You can move the PostgreSQL data on your own and the upgrade will skip this
> step if /var/opt/rh/rh-postgresql12/lib/pgsql/data/ does not exist.
> ~~
> 
> But we don't clearly mention\highlight the fact that leapp won't work
> without the directory /var/opt/rh/rh-postgresql12/lib/pgsql/data/ itself
> being present .

Correct, because it's a bug.
The fix is in https://github.com/oamg/leapp-repository/pull/1071

We can (and should) document that the folder needs to be present (and empty) for the time being.

> 
> So someone can move the data and then simply unount the
> /var/opt/rh/rh-postgresql12/lib/pgsql/data/ or delete it , and then run
> leapp which will fail with 
> 
>   File
> "/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/actors/
> satellite_upgrade_facts/actor.py", line 96, in process
>     scl_psql_stat = os.stat(POSTGRESQL_SCL_DATA_PATH)
> OSError: [Errno 2] No such file or directory:
> '/var/opt/rh/rh-postgresql12/lib/pgsql/data/'
> 
> 
> 
> So if we only consider this as a DOC BUG where we are missing to provide the
> required specifics of the manual migration, Then Yes, Even a clarification
> in the docs is good enough for me. 

I do not consider this a doc bug at all, but a code bug.
(Which we can document a workaround for)

> In that way, we may not need to deal with leapp itself but only take care of
> following the right procedure. 
> 
> Even if we consider the behavior of leapp as a bug, The first priority is to
> fix the documentation ( as that is referred by users almost everyday ) and
> then we can work on a BZ related to leapp itself.

Comment 7 Sayan Das 2023-04-21 12:12:24 UTC
MW = Maintenance window 

And yes, I agree , 
~~
We can (and should) document that the folder needs to be present (and empty) for the time being
~~

So perhaps

~~
During the upgrade, the PostgreSQL data is moved from /var/opt/rh/rh-postgresql12/lib/pgsql/data/ to /var/lib/pgsql/data/. If these two paths reside on the same partition, no further action is required. If they reside on different partitions, ensure that there is enough space for the data to be copied over. You can move the PostgreSQL data on your own and the upgrade will skip this step if /var/opt/rh/rh-postgresql12/lib/pgsql/data/ does not exist.
~~

Becomes
~~
During the upgrade, the PostgreSQL data is moved from /var/opt/rh/rh-postgresql12/lib/pgsql/data/ to /var/lib/pgsql/data/. If these two paths reside on the same partition, no further action is required. If they reside on different partitions, ensure that there is enough space for the data to be copied over. You can move the PostgreSQL data on your own and the upgrade will skip this step if /var/opt/rh/rh-postgresql12/lib/pgsql/data/ directory exists but have no content inside of it.
~~

as a part of this doc BZ fix or perhaps a bit more explanation ( if you have anything in mind )


About https://github.com/oamg/leapp-repository/pull/1071 , should i file a product bug ?


Note You need to log in before you can comment on or make changes to this bug.