Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2025948

Summary:	content prepare for 2to3 migration fails with "The requested URL's length exceeds the capacity limit for this server"
Product:	Red Hat Satellite	Reporter:	Justin Sherrill <jsherril>
Component:	Repositories	Assignee:	Justin Sherrill <jsherril>
Status:	CLOSED ERRATA	QA Contact:	Shweta Singh <shwsingh>
Severity:	medium	Docs Contact:
Priority:	high
Version:	6.9.7	CC:	aeladawy, ahumbe, fgarciad, hakon.gislason, jpasqual, ktordeur, saydas, smallamp, swadeley, vcojot, vijsingh, zhunting
Target Milestone:	6.9.8	Keywords:	Triaged
Target Release:	Unused
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	tfm-rubygem-katello-3.18.1.49-1	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-01-27 17:33:09 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Justin Sherrill 2021-11-23 13:13:48 UTC

Description of problem:

Migrating with lots of docker manifest lists (~90), will fail with "The requested URL's length exceeds the capacity limit for this server".

The user's satellite had osp13/osp16 container images synced


How reproducible:
always

Steps to Reproduce:
1.  Sync docker repos that container docker manifest lists, at least 90.
2.   run 'foreman-maintain content prepare'


Additional info:


2021-11-23T12:51:24 [E|bac|] Error message: the server returns an error
 | HTTP status code: 414
 | Response headers: {"Date"=>"Tue, 23 Nov 2021 11:51:24 GMT", "Server"=>"Apache", "Content-Length"=>"248", "Connection"=>"close", "Content-Type"=>"text/html; charset=iso-8859-1"}
 | Response body: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
 | <html><head>
 | <title>414 Request-URI Too Long</title>
 | </head><body>
 | <h1>Request-URI Too Long</h1>
 | <p>The requested URL's length exceeds the capacity
 | limit for this server.<br />
 | </p>
 | </body></html>
 |  (Pulp2to3MigrationClient::ApiError)
 | /opt/theforeman/tfm/root/usr/share/gems/gems/pulp_2to3_migration_client-0.10.0/lib/pulp_2to3_migration_client/api_client.rb:81:in `call_api'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/pulp_2to3_migration_client-0.10.0/lib/pulp_2to3_migration_client/api/pulp2_content_api.rb:139:in `list_with_http_info'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/pulp_2to3_migration_client-0.10.0/lib/pulp_2to3_migration_client/api/pulp2_content_api.rb:43:in `list'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.46/app/services/katello/pulp3/migration.rb:354:in `block in import_content_type'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:136:in `block in find_in_batches'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:238:in `block in in_batches'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:222:in `loop'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:222:in `in_batches'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:135:in `find_in_batches'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.46/app/services/katello/pulp3/migration.rb:351:in `import_content_type'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.46/app/services/katello/pulp3/migration.rb:114:in `block (4 levels) in import_pulp3_content'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.46/app/lib/katello/logging.rb:6:in `time'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.46/app/services/katello/pulp3/migration.rb:113:in `block (3 levels) in import_pulp3_content'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.46/app/services/katello/pulp3/migration.rb:112:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.46/app/services/katello/pulp3/migration.rb:112:in `block (2 levels) in import_pulp3_content'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.46/app/services/katello/pulp3/migration.rb:107:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.46/app/services/katello/pulp3/migration.rb:107:in `block in import_pulp3_content'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.46/app/lib/katello/logging.rb:6:in `time'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.46/app/services/katello/pulp3/migration.rb:106:in `import_pulp3_content'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.46/app/lib/actions/pulp3/import_migration.rb:11:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:571:in `block (3 levels) in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:32:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.46/app/lib/actions/middleware/remote_action.rb:16:in `block in run'

Comment 1 Justin Sherrill 2021-11-23 20:41:43 UTC

Workaround:


Edit /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.*/app/services/katello/pulp3/migration.rb

and change:

      GET_QUERY_ID_LENGTH = 90
to
      GET_QUERY_ID_LENGTH = 35


Then run:

satellite-maintain service restart

Comment 2 Francisco Garcia 2021-11-24 07:13:10 UTC

Hi again,

after re-running the upgrade process, it launched several hundred Refresh Distribution tasks, and none of them were progressing for 8h+ hours.

root@satellite ~ # hammer task list --search "state != stopped"                                          
-------------------------------------|-------------------------------------------------------|-----------|---------|---------------------|----------|---------------|--------------
ID                                   | ACTION                                                | STATE     | RESULT  | STARTED AT          | ENDED AT | OWNER         | TASK ERRORS
-------------------------------------|-------------------------------------------------------|-----------|---------|---------------------|----------|---------------|--------------
16e3f018-cfbd-44f7-b045-e3b2e2d38818 | InsightsCloud::Async::InsightsScheduledSync           | scheduled | pending |                     |          |               |
2e501080-7424-473a-aef3-d4930fcbab22 | Create RSS notifications                              | scheduled | pending |                     |          |               |
2e59ca5f-002d-40c8-8375-a1454f28d037 | ForemanInventoryUpload::Async::GenerateAllReportsJob  | scheduled | pending |                     |          |               |
58562523-6fc7-4501-99af-c1d57d85526e | Subscription expiration notification                  | scheduled | pending |                     |          |               | 
6921ee03-72bd-4b51-8363-873b52f5efa9 | Pulp disk space notification                          | scheduled | pending |                     |          |               | 
8d3f8f06-926d-4851-8dd7-36a7925efe8d | Inventory scheduled sync                              | scheduled | pending |                     |          | foreman_admin | 
a807834c-6579-4151-838b-128e28879dcb | Subscription Manifest validity check                  | scheduled | pending |                     |          |               | 
c80f4f03-826e-4aab-bdee-642e3d4a7ba4 | Subscription expiration notification                  | scheduled | pending |                     |          |               | 
de9b0d27-4b35-46d8-8f93-646666c7d3bf | Clean up StoredValues                                 | scheduled | pending |                     |          |               | 
edd7bbe5-373b-4ea7-8149-79d5ac708f46 | Pulp disk space notification                          | scheduled | pending |                     |          |               | 
f423c7a1-0d06-4c1e-b972-320802d75923 | Insights client status aging                          | scheduled | pending |                     |          | foreman_admin | 
b19eb5a4-f1bf-4b9a-a90c-4eb8cfa4dc7b | Refresh distribution                                  | running   | pending | 2021/11/23 14:11:08 |          | foreman_admin | 
4dc43a06-200f-4208-994f-25c68debc27f | Refresh distribution                                  | running   | pending | 2021/11/23 14:11:07 |          | foreman_admin | 
be376298-5c9d-49bd-8245-1cb289d54f76 | Refresh distribution                                  | running   | pending | 2021/11/23 14:10:59 |          | foreman_admin |
b28d938a-5799-4341-8f03-b0a2e795e8db | Refresh distribution                                  | running   | pending | 2021/11/23 14:10:58 |          | foreman_admin |
3a50255c-221b-43b3-9fd2-f5a3a17b4012 | Refresh distribution                                  | running   | pending | 2021/11/23 14:10:56 |          | foreman_admin |
61c14a7b-87e9-40ce-a13d-caa8848af41f | Refresh distribution                                  | running   | pending | 2021/11/23 14:10:55 |          | foreman_admin |
11647722-388c-4b8b-9beb-ee55ac812464 | Refresh distribution                                  | running   | pending | 2021/11/23 14:10:54 |          | foreman_admin |
caa1cb88-622b-4d78-b482-b79399ef362d | Refresh distribution                                  | running   | pending | 2021/11/23 14:10:53 |          | foreman_admin |
6a47416e-085a-46c6-8a7c-3825e5be5ca0 | Refresh distribution                                  | running   | pending | 2021/11/23 14:10:52 |          | foreman_admin |          
daa44fa1-8475-474a-b975-558faf69f28e | Refresh distribution                                  | running   | pending | 2021/11/23 14:08:54 |          | foreman_admin |              
e9a51b4c-4d89-4d73-aee6-bdcf6c806720 | Refresh distribution                                  | running   | pending | 2021/11/23 14:08:53 |          | foreman_admin |              
5521daca-f1a9-4fc4-9d07-46ad8790dedc | Refresh distribution                                  | running   | pending | 2021/11/23 14:08:52 |          | foreman_admin |              
7f7dd069-6564-4f64-bc41-8ce7ebfd6383 | Refresh distribution                                  | running   | pending | 2021/11/23 14:08:51 |          | foreman_admin |              
5270b317-1fbf-452d-acac-82159088b0c3 | Refresh distribution                                  | running   | pending | 2021/11/23 14:08:50 |          | foreman_admin |              

After it became obvious they weren't progressing I tried cleaning them, as I thought they were leftovers from an earlier upgrade process. 

root@satellite ~ # foreman-rake foreman_tasks:cleanup TASK_SEARCH="label = Actions::Pulp3::Repository::RefreshDistribution" STATES="running,paused,stopped,suspended"
API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
About to remove 693 tasks matching filter
Deleted 693 tasks matching filter
No orphaned task locks found, skipping.
No orphaned execution plans found, skipping.
No orphaned job invocations found, skipping.

Now I've relaunched the upgrade and still stuck in a task that is not progressing, Actions::Pulp3::ContentMigration :

# foreman-maintain service restart ; time foreman-maintain upgrade run --target-version=6.10 -y 
[...]
| All services started                                                [OK]      
--------------------------------------------------------------------------------
Switch support for certain content from Pulp 2 to Pulp 3: 
Performing final content migration before switching content 

# hammer task list --search 'state = running'
                                                                                                         
-------------------------------------|--------------------|---------|---------|---------------------|----------|-------------------|------------
ID                                   | ACTION             | STATE   | RESULT  | STARTED AT          | ENDED AT | OWNER             | TASK ERRORS
-------------------------------------|--------------------|---------|---------|---------------------|----------|-------------------|------------
bc656576-4717-4d6c-9f82-a76a7996385d | Content Migration  | running | pending | 2021/11/23 23:05:07 |          | foreman_api_admin |            
-------------------------------------|--------------------|---------|---------|---------------------|----------|-------------------|------------

How can I progress from here? (attached a new sosreport in the case itself)

Comment 5 Justin Sherrill 2021-12-07 18:31:11 UTC

Created redmine issue https://projects.theforeman.org/issues/34103 from this bug

Comment 7 Bryan Kearney 2021-12-13 04:05:42 UTC

Upstream bug assigned to jsherril

Comment 8 Bryan Kearney 2021-12-13 04:05:44 UTC

Upstream bug assigned to jsherril

Comment 9 Bryan Kearney 2022-01-06 00:04:59 UTC

Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/34103 has been resolved.

Comment 17 Shweta Singh 2022-01-25 16:42:04 UTC

Verified in 6.9.8 snap 2 

Synced repository containing large number of docker manifest lists(~90) 
Complete repo sync and content migration to pulp3 is successful on Satellite 6.9.8

Steps to reproduce:
1. Create a repo containing large number of docker manifest lists.
2. Sync the repo with "complete" sync.
3. Run command "foreman-maintain content prepare" to migrate content to pulp3

Expected Results: 

Migration should be successful without errors.

Actual Results:

Content Migration completed successfully.

Comment 21 errata-xmlrpc 2022-01-27 17:33:09 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Satellite 6.9.8 Async Bug Fix Update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:0320