Bug 2120632

Summary: After Upgrading LEAPPed Satellite to 6.12 pulp is not connected to redis
Product: Red Hat Satellite Reporter: Lukas Pramuk <lpramuk>
Component: UpgradesAssignee: Ewoud Kohl van Wijngaarden <ekohlvan>
Status: CLOSED ERRATA QA Contact: Lukas Pramuk <lpramuk>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.12.0CC: egolov, lvrtelov, osousa, pcreech
Target Milestone: 6.12.0Keywords: Triaged, UpgradeBlocker, Upgrades
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: satellite-installer-6.12.0.2 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-11-16 13:35:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Lukas Pramuk 2022-08-23 12:28:54 UTC
Description of problem:
After Upgrading LEAPPed Satellite 6.11 to 6.12 pulp is not connected to redis.
While upgrading *fresh* 6.11 to 6.12 doesn't reproduce the issue

Version-Release number of selected component (if applicable):
6.12.0 Snap7

How reproducible:
deterministic

Steps to Reproduce:
1. Upgrade 6.11 el7 using LEAPP to el8
2. Upgrade 6.11 el8 to 6.12
3. Check satellite health

# hammer ping
...
pulp3:            
    Status:          FAIL
    Server Response: Message: Pulp redis connection issue at https://satellite.local/pulp/api/v3.
pulp3_content:    
    Status:          ok
    Server Response: Duration: 94ms

Actual results:
there is issue with pulp connection to redis

Expected results:
pulp is connected to redis

Additional info:
GET /pulp/api/v3/status/
 
HTTP 200 OK
Allow: GET, HEAD, OPTIONS
Content-Type: application/json
Vary: Accept
 
{
    "versions": [
        {
            "component": "core",
            "version": "3.18.6"
        },
        {
            "component": "rpm",
            "version": "3.17.7"
        },
        {
            "component": "file",
            "version": "1.10.2"
        },
        {
            "component": "container",
            "version": "2.10.3"
        },
        {
            "component": "certguard",
            "version": "1.5.2"
        },
        {
            "component": "ansible",
            "version": "0.13.0"
        }
    ],
    "online_workers": [
        {
            "pulp_href": "/pulp/api/v3/workers/407fa51a-5152-4d3b-bdca-84f18e47c936/",
            "pulp_created": "2022-08-23T12:12:05.505385Z",
            "name": "29647",
            "last_heartbeat": "2022-08-23T12:17:36.171438Z",
            "current_task": null
        },
        {
            "pulp_href": "/pulp/api/v3/workers/7906df9c-2d12-480f-a269-0a2d8070ec64/",
            "pulp_created": "2022-08-23T12:12:06.517046Z",
            "name": "29679",
            "last_heartbeat": "2022-08-23T12:17:37.111536Z",
            "current_task": null
        },
        {
            "pulp_href": "/pulp/api/v3/workers/fd1836de-e44d-4e89-82c9-97390dca5b04/",
            "pulp_created": "2022-08-23T12:12:07.172465Z",
            "name": "29743",
            "last_heartbeat": "2022-08-23T12:17:37.866374Z",
            "current_task": null
        },
        {
            "pulp_href": "/pulp/api/v3/workers/62146ba9-2860-4344-a807-545f7d4501d7/",
            "pulp_created": "2022-08-23T12:12:05.829870Z",
            "name": "29615",
            "last_heartbeat": "2022-08-23T12:17:36.451599Z",
            "current_task": null
        },
        {
            "pulp_href": "/pulp/api/v3/workers/6245ecfc-1047-47fd-a5c8-4f392e54c718/",
            "pulp_created": "2022-08-23T12:12:06.913553Z",
            "name": "29711",
            "last_heartbeat": "2022-08-23T12:17:37.529098Z",
            "current_task": null
        },
        {
            "pulp_href": "/pulp/api/v3/workers/efc7b3bd-37c7-4786-b0ad-9b67f8716482/",
            "pulp_created": "2022-08-23T12:12:07.918726Z",
            "name": "29776",
            "last_heartbeat": "2022-08-23T12:17:38.503283Z",
            "current_task": null
        }
    ],
    "online_content_apps": [
        {
            "name": "29942",
            "last_heartbeat": "2022-08-23T12:17:39.118254Z"
        },
        {
            "name": "29937",
            "last_heartbeat": "2022-08-23T12:17:39.385648Z"
        },
        {
            "name": "29935",
            "last_heartbeat": "2022-08-23T12:17:39.540295Z"
        },
        {
            "name": "29946",
            "last_heartbeat": "2022-08-23T12:17:39.573142Z"
        },
        {
            "name": "29941",
            "last_heartbeat": "2022-08-23T12:17:39.573789Z"
        },
        {
            "name": "29948",
            "last_heartbeat": "2022-08-23T12:17:39.570396Z"
        },
        {
            "name": "29972",
            "last_heartbeat": "2022-08-23T12:17:39.627658Z"
        },
        {
            "name": "29951",
            "last_heartbeat": "2022-08-23T12:17:39.628058Z"
        },
        {
            "name": "29979",
            "last_heartbeat": "2022-08-23T12:17:39.649486Z"
        },
        {
            "name": "29938",
            "last_heartbeat": "2022-08-23T12:17:39.680567Z"
        },
        {
            "name": "29975",
            "last_heartbeat": "2022-08-23T12:17:39.744316Z"
        },
        {
            "name": "29963",
            "last_heartbeat": "2022-08-23T12:17:39.816222Z"
        },
        {
            "name": "29981",
            "last_heartbeat": "2022-08-23T12:17:39.823926Z"
        }
    ],
    "database_connection": {
        "connected": true
    },
    "redis_connection": {
        "connected": false
    },
    "storage": {
        "total": 536859357184,
        "used": 181628755968,
        "free": 355230601216
    }
}

Comment 1 Evgeni Golov 2022-08-23 14:15:22 UTC
FTR, the issue is not as much that it was leapp'ed, but that it was upgraded from an older release.

Comment 2 Evgeni Golov 2022-08-23 14:20:45 UTC
workaround: foreman-installer --foreman-proxy-content-pulpcore-cache-enabled true

Comment 3 Bryan Kearney 2022-08-23 16:05:31 UTC
Upstream bug assigned to ekohlvan

Comment 4 Bryan Kearney 2022-08-23 16:05:32 UTC
Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/35157 has been resolved.

Comment 5 Lukas Pramuk 2022-08-24 12:18:32 UTC
To work around missing migration just run: 

# satellite-installer --foreman-proxy-content-pulpcore-cache-enabled true

Comment 6 Lukas Pramuk 2022-09-08 08:46:38 UTC
What rpm actually fixes the bug ?  
I see an empty Fixed In Version field

Comment 7 Evgeni Golov 2022-09-08 08:53:58 UTC
The fix is in the right branch (https://gitlab.cee.redhat.com/satellite/satellite-installer/-/commit/7fa34a483bcc2ed13a21e230901d53163255d602) but I don't see a build of satellite-installer-6.12.0.2 in brew

Comment 9 Lukas Pramuk 2022-09-21 15:31:26 UTC
VERIFIED.

@Satellite 6.12.0 Snap11

by the following reproducer:
1) Upgrade 6.11 el8 to 6.12
2) Check satellite health

# hammer ping
...

pulp3:            
    Status:          ok
    Server Response: Duration: 474ms
pulp3_content:    
    Status:          ok
    Server Response: Duration: 100ms

>>> pulp3 component has status ok

Comment 13 errata-xmlrpc 2022-11-16 13:35:34 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.12 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:8506