RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2144304 - [RFE] leapp should highlight the generated report and possible issues better so people not overlook it
Summary: [RFE] leapp should highlight the generated report and possible issues better ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: leapp-repository
Version: 8.7
Hardware: x86_64
OS: Linux
unspecified
low
Target Milestone: rc
: ---
Assignee: Petr Stodulka
QA Contact: Martin Klusoň
URL:
Whiteboard:
Depends On: 2223312
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-11-20 15:54 UTC by Vinícius Ferrão
Modified: 2023-11-14 17:01 UTC (History)
4 users (show)

Fixed In Version: leapp-repository-0.18.0-5.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-11-14 15:35:03 UTC
Type: Bug
Target Upstream Version:
Embargoed:
pm-rhel: mirror+


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker OAMG-7930 0 None None None 2022-11-20 15:56:02 UTC
Red Hat Issue Tracker RHELPLAN-139977 0 None None None 2022-11-20 15:56:00 UTC
Red Hat Product Errata RHBA-2023:7013 0 None None None 2023-11-14 15:35:36 UTC

Description Vinícius Ferrão 2022-11-20 15:54:04 UTC
Description of problem:
leapp does not check if the server is running pgsql with impossible upgrade path. So when you run leapp on a EL8 server to EL9 postgres fails to start after the upgrade and it stays on a broken state since you probably aren't running pgsql 12 on EL8 system.

postgresql-check-db-dir[833]: An old version '10' of the database format was found.
postgresql-check-db-dir[833]: You need to dump and reload before using PostgreSQL 13
systemd[1]: postgresql.service: Control process exited, code=exited, status=1/FAILURE
systemd[1]: postgresql.service: Failed with result 'exit-code'.
systemd[1]: Failed to start PostgreSQL database server.

As example, leapp checks for unsupported FreeIPA updates:

Upgrade has been inhibited due to the following problems:
    1. Inhibitor: ipa-server does not support in-place upgrade

It should do something similar with Postgres.

Version-Release number of selected component (if applicable):
leapp-0.15.0-2.el8.noarch
leapp-deps-0.15.0-2.el8.noarch
python3-leapp-0.15.0-2.el8.noarch
leapp-upgrade-el8toel9-deps-0.17.0-3.el8.noarch
leapp-upgrade-el8toel9-0.17.0-3.el8.noarch

How reproducible:
100%

Steps to Reproduce:
1. Upgrade a server with Postgres <= 11 to EL9

Actual results:
Not a single warning during the upgrade, which may end up with broken servers after upgrade.

Expected results:
A warning should be displayed with instructions to properly upgrade the server, for example pointing to the release file: /usr/share/doc/postgresql/README.rpm-dist

Additional info:
N/A

Comment 1 Honza Horak 2022-12-06 13:56:14 UTC
There is a warning that is supposed to be presented when postgresql-server is installed:

https://github.com/oamg/leapp-repository/blob/master/repos/system_upgrade/el8toel9/actors/postgresqlcheck/libraries/postgresqlcheck.py#L8_L11

Maybe there is a problem with the detection? Anyway, let's try to reproduce and see what happens.

Comment 2 Honza Horak 2022-12-06 15:42:16 UTC
I tried it myself and just after installing the postgresql-server package (even not enabled), I read in the report:

----------------------------------------
Risk Factor: medium
Title: PostgreSQL (postgresql-server) has been detected on your system
Summary: PostgreSQL server component will be upgraded. Since RHEL-9 includes PostgreSQL server 13 by default, which is incompatible with 9.6, 10 and 12 included in RHEL-8, it is necessary to proceed with additional steps for the complete upgrade of the PostgreSQL data.
Remediation: [hint] Back up your data before proceeding with the upgrade and follow steps in the documentation section "Migrating to a RHEL 9 version of PostgreSQL" after the upgrade.
Key: 6966486bbbf49710d1d9f4e4f6b612217aca38a1
----------------------------------------

Then, navigating myself to the "4.7. Migrating to a RHEL 9 version of PostgreSQL" section at
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html-single/configuring_and_using_database_servers/index#migrating-to-a-rhel-9-version-of-postgresql_using-postgresql

...I read:

"""
...
 The following procedure describes migration from the RHEL 8 version of PostgreSQL 12 to the RHEL 9 version of PostgreSQL 13 using the fast upgrade method. For migration from postgresql streams other than 12, use one of the following approaches:

* Update your PostgreSQL server to version 12 on RHEL 8 and then use the pg_upgrade utility to perform the fast upgrade to RHEL 9 version of PostgreSQL 13.

* Use the dump and restore upgrade directly between any RHEL 8 version of PostgreSQL and PostgreSQL 13 in RHEL 9.
...
"""

From my perspective, this gives pretty clear guidelines what to do. If there is anything not clear or misleading, please, provide more input to us, so we can improve the documentation.

The Leapp pre-upgrade output is deliberately short and directs to a more verbose documentation, I don't think this should change, because it would make the Leapp output unreadable.

Comment 3 Vinícius Ferrão 2022-12-06 17:18:00 UTC
Hi Honza, thanks for the reply. I can try to do it again here, but as far as I understood the Risk factor is medium, right? Shouldn't it be a block? Because the medium risk factor does not impede the upgrade, and for those who aren't paying attention it may be an issue.

The warning seems to have been popped here but I didn't see it because I wasn't paying attention:

leapp-report.json:            "title": "Migrating to a RHEL 9 version of PostgreSQL",
leapp-report.json:            "title": "postgresql-server"
leapp-report.json:            "context": "Back up your data before proceeding with the upgrade and follow steps in the documentation section \"Migrating to a RHEL 9 version of PostgreSQL\" after the upgrade.",
leapp-report.json:      "summary": "PostgreSQL server component will be upgraded. Since RHEL-9 includes PostgreSQL server 13 by default, which is incompatible with 9.6, 10 and 12 included in RHEL-8, it is necessary to proceed with additional steps for the complete upgrade of the PostgreSQL data.",
leapp-report.json:      "title": "PostgreSQL (postgresql-server) has been detected on your system",
leapp-report.json:      "actor": "postgresql_check",
leapp-report.txt:Title: PostgreSQL (postgresql-server) has been detected on your system
leapp-report.txt:Summary: PostgreSQL server component will be upgraded. Since RHEL-9 includes PostgreSQL server 13 by default, which is incompatible with 9.6, 10 and 12 included in RHEL-8, it is necessary to proceed with additional steps for the complete upgrade of the PostgreSQL data.
leapp-report.txt:Remediation: [hint] Back up your data before proceeding with the upgrade and follow steps in the documentation section "Migrating to a RHEL 9 version of PostgreSQL" after the upgrade.


Can I recommend the warning to be changed to something that is like a block, but you can override? Just like VDO? Where I must explicitly do leapp answer --section check_vdo.no_vdo_devices=True to proceed?

What do you think?

Thanks.

Comment 4 Honza Horak 2022-12-14 17:27:04 UTC
I'm actually not the correct one to speak about the proper severity, this might be the choice of the Leapp folks, because it must be in line with other reports. I'll let Petr or somebody else speak about this.

Comment 5 Petr Stodulka 2023-01-04 13:50:37 UTC
Hi everyone, thank you for the feedback. Let me first explain the current situation/design and go through particular cases.

-----
Raising the severity does not cause stop of the upgrade. The upgrade is stopped only when the report specify it's an inhibitor - and inhibitors are expected to have set the high severity. Inhibitors cannot be skipped nor overwritten.
 
Risk factors are defined in the official documentation https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/upgrading_from_rhel_7_to_rhel_8/index#assessing-upgradability-and-applying-automated-remediations-through-the-web-console_upgrading-from-rhel-7-to-rhel-8:
* High - very likely to result in a deteriorated system state
* Medium - can impact both the system and applications
* Low - should not impact the system but can have an impact on applications
* Info - informational with no expected impact to the system or applications

From this point, I would say it's correctly medium. The system is bootable, accessible and user can work on fixing all issues. Also, reading the generated report is mandatory step and every user is required to consider if they want to continue in the upgrade or not after reading the report. That's the reason, why the procedure of the upgrade is:
 - back up the system
 - # leapp preupgrade
 - read the generated report
 - # leapp upgrade
(back up could be done just right before the upgrade, preupgrade is not affecting the system).

In case of VDO the situation is very different from the postgresql one. In case of postgresql, all instructions (what to do) are provided and system is running / bootable after the upgrade. In case of VDO the impact is bigger and system could become unbootable in some situations - next to the point that data are not accessible anymore if any uncoverted VDO partition has been present during the upgrade. Also in case of VDO we are not always able to check all partitions whether they are safe for the upgrade or not. For that purpose, in such situations we ask users to decide the state of the system.

In case of IPA server, the upgrade is not supported at all and has another consequences. So it's again a different situation from this one.

---------

From this point, this ticket could be closed, however I want to keep it opened at least for now if nobody is against, so we could think about a possible improvement as I understand that this problem happened and possibly it will happen to other users as well (not only in case of postgresql, we have additional applications that could lead to similar problems and the current report generated by postgresql is correct regarding our all expectations). I want to be clear, that we do not want to overuse raising of user-questions during the upgrade and current policy is to use the user question only when we really need user to decide something what we can't.

From my understanding of this ticket (correct me if I am wrong), the report has not been read prior the upgrade. If this was the case, I am opened to suggestion how to highlight the requirement for the read of the report better than we do now, in non-blocking way. Or if you read the generated report file, have you skipped some lower priorities reports and focused just on high risk factors? Also in such a case, have you read the official upgrade documentation? If not, did you know about the official upgrade documentation? I am asking for the purpose to identify better what is the actual problem as I would like to understand what we could actually improve. I will raise also the discussion about possible future changes in our policies and what we could do to improve the experience. Thank you for the feedback.

Comment 6 Vinícius Ferrão 2023-01-11 04:23:32 UTC
Hi @pstodulk you're correct that I didn't read the warning. Nothing grabbed my attention during the run (execution) to be honest. I think I ran leapp upgrade without constantly check the terminal and just rebooted after it finished. So I think it should warn at the end or something else to catch our attention, red font, blinking text, you name it.

I usually don't do this kind of sh*t, but mistakes happens. I was in an end of year rush and wanted to do some tests before the holidays.

Regarding the docs issue, I read it on access.redhat.com and again, nothing showed up alarming as it would caught my attention regarding database upgrades. But I think the docs are OK, if I would blame anyone else instead of me would be the console output of leapp the didn't get my attention.

Also a rollback/cancel is missing on leapp. But that's another issue.

Thanks.

Comment 7 Petr Stodulka 2023-01-11 11:58:45 UTC
Thank you Vinícius for the feedback, I appreciate it. About the rollback, that's currently possibly only in some scenarious, using the Boom tool, but we do not plan right now to cover this option in the tooling (or the upgrade process) itself due to all possible problems & limitations connected to such rollbacks. I agree with you that we should improve the console output so we catch users' attention and I understand that it's probably pretty easy to miss that especially when people are in rush to finish a stuff. We will discuss what could be the best solution here. Thank you. I am updating the title and keeping this ticket opened for the future tracking.

Comment 10 Petr Stodulka 2023-07-17 20:01:24 UTC
The change have been delivered in upstream PRs:
* https://github.com/oamg/leapp/pull/818
* https://github.com/oamg/leapp-repository/pull/1061

Depends on bug 2223312.

Comment 17 errata-xmlrpc 2023-11-14 15:35:03 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (leapp-repository bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:7013


Note You need to log in before you can comment on or make changes to this bug.