Description of problem: On Red Hat Satellite 6.12 removing orphans task going to the paused state ----------------------------------------------------- # foreman-rake katello:delete_orphaned_content RAILS_ENV=production >/dev/null Error message: the server returns an error HTTP status code: 500 Response headers: {"Date"=>"Wed, 04 Jan 2023 09:18:12 GMT", "Server"=>"gunicorn", "Content-Type"=>"text/html; charset=UTF-8", "X-Frame-Options"=>"DENY", "Content-Length"=>"145", "Vary"=>"Cookie", "X-Content-Type-Options"=>"nosniff", "Referrer-Policy"=>"same-origin", "Correlation-ID"=>"877956a7c9a64de69fa0726e90fabfed", "Access-Control-Expose-Headers"=>"Correlation-ID", "Via"=>"1.1 exrhel0050.it.rm.dk", "Connection"=>"close"} Response body: <!doctype html> <html lang="en"> <head> <title>Server Error (500)</title> </head> <body> <h1>Server Error (500)</h1><p></p> </body> </html> -------------------------------------------- /var/log/foreman/production.log 2023-01-04T10:18:12 [E|bac|] Error message: the server returns an error | <title>Server Error (500)</title> | <h1>Server Error (500)</h1><p></p> | (PulpRpmClient::ApiError) | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:483:in `block in with_error_handling' | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:483:in `with_error_handling' 2023-01-04T10:18:12 [I|bac|] Task {label: Actions::Katello::OrphanCleanup::RemoveOrphans, id: 900191f2-f3e2-440b-b572-c4e864928580, execution_plan_id: d62d2d8b-18ba-4b42-912a-cd85b8194604} state changed: paused result: error -------------------------------------------- # foreman-rake katello:delete_orphaned_content RAILS_ENV=production >/dev/null fails and go into pause state. # journalctl --since "1 hour ago" -u pulpcore-api Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: pulp [b811b69754d74a4995ec4ab8a24b4e1b]: - - [05/Jan/2023:10:15:35 +0000] "GET /pulp/api/v3/repositories/rpm/rpm/85> Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: pulp [fa2ca280585f42a2a59dd4c64fdbbe88]: - - [05/Jan/2023:10:15:35 +0000] "GET /pulp/api/v3/repositories/rpm/rpm/1f> Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: pulp [389168a01feb4386afc5673af8564dae]: - - [05/Jan/2023:10:15:35 +0000] "GET /pulp/api/v3/repositories/rpm/rpm/1c> Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: pulp [ca7f02dd5dc94c6293f31bea06589ee5]: django.request:ERROR: Internal Server Error: /pulp/api/v3/repositories/rpm/> Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: Traceback (most recent call last): Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: File "/usr/lib/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: response = get_response(request) Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: File "/usr/lib/python3.9/site-packages/django/core/handlers/base.py", line 181, in _get_response Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: response = wrapped_callback(request, *callback_args, **callback_kwargs) Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: File "/usr/lib/python3.9/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: return view_func(*args, **kwargs) Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: File "/usr/lib/python3.9/site-packages/rest_framework/viewsets.py", line 125, in view Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: return self.dispatch(request, *args, **kwargs) Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: File "/usr/lib/python3.9/site-packages/rest_framework/views.py", line 509, in dispatch Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: response = self.handle_exception(exc) Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: File "/usr/lib/python3.9/site-packages/rest_framework/views.py", line 469, in handle_exception Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: self.raise_uncaught_exception(exc) Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: File "/usr/lib/python3.9/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: raise exc Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: File "/usr/lib/python3.9/site-packages/rest_framework/views.py", line 506, in dispatch Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: response = handler(request, *args, **kwargs) Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: File "/usr/lib/python3.9/site-packages/rest_framework/mixins.py", line 43, in list Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: return self.get_paginated_response(serializer.data) Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: File "/usr/lib/python3.9/site-packages/rest_framework/serializers.py", line 768, in data Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: ret = super().data Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: File "/usr/lib/python3.9/site-packages/rest_framework/serializers.py", line 253, in data Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: self._data = self.to_representation(self.instance) Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: File "/usr/lib/python3.9/site-packages/rest_framework/serializers.py", line 686, in to_representation Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: return [ Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: File "/usr/lib/python3.9/site-packages/rest_framework/serializers.py", line 687, in <listcomp> Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: self.child.to_representation(item) for item in iterable Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: File "/usr/lib/python3.9/site-packages/rest_framework/serializers.py", line 522, in to_representation Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: ret[field.field_name] = field.to_representation(attribute) Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: File "/usr/lib/python3.9/site-packages/pulpcore/app/serializers/repository.py", line 371, in to_representation Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: item_dict = {"count": count_detail.count, "href": count_detail.content_href} Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: File "/usr/lib/python3.9/site-packages/pulpcore/app/models/repository.py", line 1118, in content_href Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: ctype_model = ctype_query.first().cast().__class__ Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: AttributeError: 'NoneType' object has no attribute 'cast' Jan 05 11:15:35 exrhel0050.it.rm.dk pulpcore-api[1693]: pulp [ca7f02dd5dc94c6293f31bea06589ee5]: - - [05/Jan/2023:10:15:35 +0000] "GET /pulp/api/v3/repositories/rpm/rpm/58> lines 164-210/210 (END) ----------------------------------------------------- Additional info: Satellite running version 6.12.0 on RHEL8.7. Was upgraded start december 2022 from version 6.11.4
I believe this is fixed upstream in #3299 - the fix needs to be backported.
Backporting and releasing upstream
Verified. Version Tested: Satellite 6.13 Snap 13.0 Verification Steps: 1. Import a manifest and enable few repositories on satellite(preferably medium to large repos). 2. Start syncing repositories. 3. Remove the enabled repositories while it is syncing. 4. Delete the orphaned repositories using command "foreman-rake katello:delete_orphaned_content RAILS_ENV=production >/dev/null". Result: All the orphaned repositories will be removed.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Satellite 6.13 Release), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2023:2097