+++ This bug was initially created as a clone of Bug #2160008 +++
Description of problem:
On Red Hat Satellite 6.12 removing orphans task going to the paused state
-----------------------------------------------------
# foreman-rake katello:delete_orphaned_content RAILS_ENV=production >/dev/null
Error message: the server returns an error
HTTP status code: 500
Response headers: {"Date"=>"Wed, 04 Jan 2023 09:18:12 GMT", "Server"=>"gunicorn", "Content-Type"=>"text/html; charset=UTF-8", "X-Frame-Options"=>"DENY", "Content-Length"=>"145", "Vary"=>"Cookie", "X-Content-Type-Options"=>"nosniff", "Referrer-Policy"=>"same-origin", "Correlation-ID"=>"877956a7c9a64de69fa0726e90fabfed", "Access-Control-Expose-Headers"=>"Correlation-ID", "Via"=>"1.1 exrhel0050.it.rm.dk", "Connection"=>"close"}
Response body:
<!doctype html>
<html lang="en">
<head>
<title>Server Error (500)</title>
</head>
<body>
<h1>Server Error (500)</h1><p></p>
</body>
</html>
--------------------------------------------
/var/log/foreman/production.log
2023-01-04T10:18:12 [E|bac|] Error message: the server returns an error
| <title>Server Error (500)</title>
| <h1>Server Error (500)</h1><p></p>
| (PulpRpmClient::ApiError)
| /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:483:in `block in with_error_handling'
| /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:483:in `with_error_handling'
2023-01-04T10:18:12 [I|bac|] Task {label: Actions::Katello::OrphanCleanup::RemoveOrphans, id: 900191f2-f3e2-440b-b572-c4e864928580, execution_plan_id: d62d2d8b-18ba-4b42-912a-cd85b8194604} state changed: paused result: error
--------------------------------------------
# foreman-rake katello:delete_orphaned_content RAILS_ENV=production >/dev/null
fails and go into pause state.
# journalctl --since "1 hour ago" -u pulpcore-api
Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: pulp [b811b69754d74a4995ec4ab8a24b4e1b]: - - [05/Jan/2023:10:15:35 +0000] "GET /pulp/api/v3/repositories/rpm/rpm/85>
Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: pulp [fa2ca280585f42a2a59dd4c64fdbbe88]: - - [05/Jan/2023:10:15:35 +0000] "GET /pulp/api/v3/repositories/rpm/rpm/1f>
Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: pulp [389168a01feb4386afc5673af8564dae]: - - [05/Jan/2023:10:15:35 +0000] "GET /pulp/api/v3/repositories/rpm/rpm/1c>
Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: pulp [ca7f02dd5dc94c6293f31bea06589ee5]: django.request:ERROR: Internal Server Error: /pulp/api/v3/repositories/rpm/>
Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: Traceback (most recent call last):
Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: File "/usr/lib/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: response = get_response(request)
Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: File "/usr/lib/python3.9/site-packages/django/core/handlers/base.py", line 181, in _get_response
Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: response = wrapped_callback(request, *callback_args, **callback_kwargs)
Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: File "/usr/lib/python3.9/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: return view_func(*args, **kwargs)
Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: File "/usr/lib/python3.9/site-packages/rest_framework/viewsets.py", line 125, in view
Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: return self.dispatch(request, *args, **kwargs)
Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: File "/usr/lib/python3.9/site-packages/rest_framework/views.py", line 509, in dispatch
Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: response = self.handle_exception(exc)
Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: File "/usr/lib/python3.9/site-packages/rest_framework/views.py", line 469, in handle_exception
Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: self.raise_uncaught_exception(exc)
Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: File "/usr/lib/python3.9/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: raise exc
Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: File "/usr/lib/python3.9/site-packages/rest_framework/views.py", line 506, in dispatch
Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: response = handler(request, *args, **kwargs)
Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: File "/usr/lib/python3.9/site-packages/rest_framework/mixins.py", line 43, in list
Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: return self.get_paginated_response(serializer.data)
Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: File "/usr/lib/python3.9/site-packages/rest_framework/serializers.py", line 768, in data
Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: ret = super().data
Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: File "/usr/lib/python3.9/site-packages/rest_framework/serializers.py", line 253, in data
Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: self._data = self.to_representation(self.instance)
Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: File "/usr/lib/python3.9/site-packages/rest_framework/serializers.py", line 686, in to_representation
Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: return [
Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: File "/usr/lib/python3.9/site-packages/rest_framework/serializers.py", line 687, in <listcomp>
Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: self.child.to_representation(item) for item in iterable
Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: File "/usr/lib/python3.9/site-packages/rest_framework/serializers.py", line 522, in to_representation
Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: ret[field.field_name] = field.to_representation(attribute)
Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: File "/usr/lib/python3.9/site-packages/pulpcore/app/serializers/repository.py", line 371, in to_representation
Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: item_dict = {"count": count_detail.count, "href": count_detail.content_href}
Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: File "/usr/lib/python3.9/site-packages/pulpcore/app/models/repository.py", line 1118, in content_href
Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: ctype_model = ctype_query.first().cast().__class__
Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: AttributeError: 'NoneType' object has no attribute 'cast'
Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: pulp [ca7f02dd5dc94c6293f31bea06589ee5]: - - [05/Jan/2023:10:15:35 +0000] "GET /pulp/api/v3/repositories/rpm/rpm/58>
lines 164-210/210 (END)
-----------------------------------------------------
Additional info:
Satellite running version 6.12.0 on RHEL8.7. Was upgraded start december 2022 from version 6.11.4
--- Additional comment from on 2023-01-18T18:58:22Z
I believe this is fixed upstream in #3299 - the fix needs to be backported.
--- Additional comment from on 2023-01-18T22:46:44Z
Backporting and releasing upstream
Verified.
Version Tested: Satellite 6.12.3 Snap 1.0
Verification Steps:
1. Import a manifest and enable few repositories on satellite(preferably medium to large repos).
2. Start syncing repositories.
3. Remove the enabled repositories while it is syncing.
4. Delete the orphaned repositories using command "foreman-rake katello:delete_orphaned_content RAILS_ENV=production >/dev/null".
Result:
All the orphaned repositories will be removed.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (Important: Satellite 6.12.3 Async Security Update), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHSA-2023:1630