Bug 1333110 - Uploading rpm fails with "InvalidStringData: strings in documents must be valid UTF-8"
Summary: Uploading rpm fails with "InvalidStringData: strings in documents must be val...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Pulp
Version: 6.1.8
Hardware: x86_64
OS: Linux
urgent
urgent vote
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: jcallaha
URL:
Whiteboard:
: 1392452 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-05-04 16:35 UTC by anerurka
Modified: 2019-06-13 21:25 UTC (History)
20 users (show)

Fixed In Version: pulp-rpm-2.8.7.16-1
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1480345 (view as bug list)
Environment:
Last Closed: 2017-09-25 18:59:44 UTC


Attachments (Terms of Use)
verification screenshot 1 (58.71 KB, image/png)
2017-08-09 20:11 UTC, jcallaha
no flags Details
verification screenshot 2 (58.71 KB, image/png)
2017-08-09 20:11 UTC, jcallaha
no flags Details
non-utf8 rpm (6.31 KB, application/x-rpm)
2017-08-28 15:49 UTC, jcallaha
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:2803 normal SHIPPED_LIVE Satellite 6.2.12 bug fix update 2017-10-12 19:22:49 UTC
Pulp Redmine 1903 Normal CLOSED - CURRENTRELEASE RPM import traceback (non-utf-8 metadata slipping through) 2017-04-19 21:35:10 UTC
Pulp Redmine 2622 High CLOSED - CURRENTRELEASE Sync fails when non-ASCII characters are present in primary.xml 2017-08-15 18:32:27 UTC

Description anerurka 2016-05-04 16:35:44 UTC
Description of problem:

Uploading rpm fails with "InvalidStringData: strings in documents must be valid UTF-8"

Version-Release number of selected component (if applicable):

Satellite Server v.6.1.x

How reproducible:

Steps to Reproduce:

1. Upload a rpm into the Satellite Server which is not UTF-8 compliant

Login to Satellite --> Content Product --> Select the Product --> Select a Repository --> Upload the RPM

2. Task will show as success.

3. However RPM is not uploaded in the repository and below error is capatured in the "/var/log/messages"

~~~~~
InvalidStringData: strings in documents must be valid UTF-8
~~~~~

Actual results:

RPM is not uploaded in the repository


Expected results:

RPM uploaded successfully and visible in the repository

Additional info:

>> Is there any easy way to do a sanity check on the rpm-file, so that problem can be verified?

>> And still, why is Satellite reporting success, when the foreman-log shows error?

Comment 3 Michael Hrivnak 2016-06-13 02:10:34 UTC
I don't think this is a regression. I think this has been the behavior for several pulp releases. I'm linking an upstream bug where we're tracking the issue.

Comment 4 Bryan Kearney 2016-07-26 18:58:39 UTC
Moving 6.2 bugs out to sat-backlog.

Comment 6 pulp-infra@redhat.com 2016-11-21 18:54:27 UTC
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 7 pulp-infra@redhat.com 2016-11-21 18:54:30 UTC
The Pulp upstream bug priority is at Low. Updating the external tracker on this bug.

Comment 8 Brian Bouterse 2016-11-21 18:58:15 UTC
*** Bug 1392452 has been marked as a duplicate of this bug. ***

Comment 9 pulp-infra@redhat.com 2016-12-09 22:02:18 UTC
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.

Comment 10 pulp-infra@redhat.com 2017-02-20 03:02:58 UTC
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 11 pulp-infra@redhat.com 2017-02-20 03:03:01 UTC
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

Comment 12 pulp-infra@redhat.com 2017-02-22 16:03:29 UTC
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.

Comment 13 pulp-infra@redhat.com 2017-02-23 00:03:20 UTC
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.

Comment 14 pulp-infra@redhat.com 2017-02-24 22:03:05 UTC
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

Comment 15 pulp-infra@redhat.com 2017-02-24 22:33:10 UTC
All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.

Comment 16 Dennis Kliban 2017-03-14 02:28:30 UTC
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.

Comment 17 pulp-infra@redhat.com 2017-03-21 19:19:53 UTC
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

Comment 18 pulp-infra@redhat.com 2017-03-21 19:34:01 UTC
All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.

Comment 19 pulp-infra@redhat.com 2017-04-05 19:05:44 UTC
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.

Comment 22 pulp-infra@redhat.com 2017-04-19 21:35:12 UTC
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Comment 25 Chris Duryee 2017-07-07 19:31:52 UTC
Created attachment 1295394 [details]
pulp-rpm-common hotfix rpm

Comment 34 jcallaha 2017-08-09 20:10:11 UTC
Verified in Satellite 6.3 Snap 10

Uploading rpms now works with both yum and file type repositories. See attached images for uploaded files.

--- rpm repo ---
2017-08-09 15:30:32 a4d731f9 [app] [I] Started POST "/katello/api/v2/repositories/19/upload_content" for 10.13.129.211 at 2017-08-09 15:30:32 -0400
2017-08-09 15:30:32 a4d731f9 [app] [I] Processing by Katello::Api::V2::RepositoriesController#upload_content as HTML
2017-08-09 15:30:32 a4d731f9 [app] [I]   Parameters: {"content"=>[#<ActionDispatch::Http::UploadedFile:0x00000011c756f8 @tempfile=#<Tempfile:/tmp/RackMultipart20170809-23802-11q9006.rpm>, @original_filename="facter-2.4.6-3.el7sat.x86_64.rpm", @content_type="application/x-rpm", @headers="Content-Disposition: form-data; name=\"content[]\"; filename=\"facter-2.4.6-3.el7sat.x86_64.rpm\"\r\nContent-Type: application/x-rpm\r\n">, #<ActionDispatch::Http::UploadedFile:0x00000011c756d0 @tempfile=#<Tempfile:/tmp/RackMultipart20170809-23802-15gxn47.rpm>, @original_filename="gofer-2.7.6-1.el7sat.noarch.rpm", @content_type="application/x-rpm", @headers="Content-Disposition: form-data; name=\"content[]\"; filename=\"gofer-2.7.6-1.el7sat.noarch.rpm\"\r\nContent-Type: application/x-rpm\r\n">, #<ActionDispatch::Http::UploadedFile:0x00000011c756a8 @tempfile=#<Tempfile:/tmp/RackMultipart20170809-23802-h3n0t5.rpm>, @original_filename="hiera-1.3.1-2.el7sat.noarch.rpm", @content_type="application/x-rpm", @headers="Content-Disposition: form-data; name=\"content[]\"; filename=\"hiera-1.3.1-2.el7sat.noarch.rpm\"\r\nContent-Type: application/x-rpm\r\n">, #<ActionDispatch::Http::UploadedFile:0x00000011c75680 @tempfile=#<Tempfile:/tmp/RackMultipart20170809-23802-1pwferp.rpm>, @original_filename="katello-agent-2.5.0-5.el7sat.noarch.rpm", @content_type="application/x-rpm", @headers="Content-Disposition: form-data; name=\"content[]\"; filename=\"katello-agent-2.5.0-5.el7sat.noarch.rpm\"\r\nContent-Type: application/x-rpm\r\n">, #<ActionDispatch::Http::UploadedFile:0x00000011c75658 @tempfile=#<Tempfile:/tmp/RackMultipart20170809-23802-1ov8u5f.rpm>, @original_filename="katello-agent-2.9.0-2.el7sat.noarch.rpm", @content_type="application/x-rpm", @headers="Content-Disposition: form-data; name=\"content[]\"; filename=\"katello-agent-2.9.0-2.el7sat.noarch.rpm\"\r\nContent-Type: application/x-rpm\r\n">, #<ActionDispatch::Http::UploadedFile:0x00000011c75630 @tempfile=#<Tempfile:/tmp/RackMultipart20170809-23802-gf7yds.rpm>, @original_filename="libtasn1-3.8-3.el7.x86_64.rpm", @content_type="application/x-rpm", @headers="Content-Disposition: form-data; name=\"content[]\"; filename=\"libtasn1-3.8-3.el7.x86_64.rpm\"\r\nContent-Type: application/x-rpm\r\n">, #<ActionDispatch::Http::UploadedFile:0x00000011c75608 @tempfile=#<Tempfile:/tmp/RackMultipart20170809-23802-bnjatj.rpm>, @original_filename="pulp-puppet-tools-2.8.7.1-1.el7sat.noarch.rpm", @content_type="application/x-rpm", @headers="Content-Disposition: form-data; name=\"content[]\"; filename=\"pulp-puppet-tools-2.8.7.1-1.el7sat.noarch.rpm\"\r\nContent-Type: application/x-rpm\r\n">, #<ActionDispatch::Http::UploadedFile:0x00000011c755e0 @tempfile=#<Tempfile:/tmp/RackMultipart20170809-23802-1krvsgt.rpm>, @original_filename="pulp-rpm-handlers-2.8.7.5-1.el7sat.noarch.rpm", @content_type="application/x-rpm", @headers="Content-Disposition: form-data; name=\"content[]\"; filename=\"pulp-rpm-handlers-2.8.7.5-1.el7sat.noarch.rpm\"\r\nContent-Type: application/x-rpm\r\n">, #<ActionDispatch::Http::UploadedFile:0x00000011c755b8 @tempfile=#<Tempfile:/tmp/RackMultipart20170809-23802-11ord10.rpm>, @original_filename="puppet-3.8.6-2.el7sat.noarch.rpm", @content_type="application/x-rpm", @headers="Content-Disposition: form-data; name=\"content[]\"; filename=\"puppet-3.8.6-2.el7sat.noarch.rpm\"\r\nContent-Type: application/x-rpm\r\n">, #<ActionDispatch::Http::UploadedFile:0x00000011c75590 @tempfile=#<Tempfile:/tmp/RackMultipart20170809-23802-6at6oa.rpm>, @original_filename="python-gofer-2.7.6-1.el7sat.noarch.rpm", @content_type="application/x-rpm", @headers="Content-Disposition: form-data; name=\"content[]\"; filename=\"python-gofer-2.7.6-1.el7sat.noarch.rpm\"\r\nContent-Type: application/x-rpm\r\n">, #<ActionDispatch::Http::UploadedFile:0x00000011c75568 @tempfile=#<Tempfile:/tmp/RackMultipart20170809-23802-1pvohg4.rpm>, @original_filename="python-gofer-proton-2.7.6-1.el7sat.noarch.rpm", @content_type="application/x-rpm", @headers="Content-Disposition: form-data; name=\"content[]\"; filename=\"python-gofer-proton-2.7.6-1.el7sat.noarch.rpm\"\r\nContent-Type: application/x-rpm\r\n">], "authenticity_token"=>"dfRAgSrUNzDxLBarDLymIy1ET9x5XkPAavna51fwRn2NAz4cF4P1SJEmo4QcmKLuWN51GGutn3bY4XXPtsBEeQ==", "api_version"=>"v2", "id"=>"19"}
2017-08-09 15:30:32 a4d731f9 [app] [I] Current user: admin (administrator)
2017-08-09 15:30:37 a4d731f9 [app] [I] Completed 200 OK in 4799ms (Views: 0.4ms | ActiveRecord: 20.7ms)


--- file repo ---
2017-08-09 15:41:25 394054ad [app] [I] Started POST "/katello/api/v2/repositories/20/upload_content" for 10.13.129.211 at 2017-08-09 15:41:25 -0400
2017-08-09 15:41:25 394054ad [app] [I] Processing by Katello::Api::V2::RepositoriesController#upload_content as HTML
2017-08-09 15:41:25 394054ad [app] [I]   Parameters: {"content"=>[#<ActionDispatch::Http::UploadedFile:0x007f5363a7e758 @tempfile=#<Tempfile:/tmp/RackMultipart20170809-23802-2576oi.rpm>, @original_filename="facter-2.4.6-3.el7sat.x86_64.rpm", @content_type="application/x-rpm", @headers="Content-Disposition: form-data; name=\"content[]\"; filename=\"facter-2.4.6-3.el7sat.x86_64.rpm\"\r\nContent-Type: application/x-rpm\r\n">, #<ActionDispatch::Http::UploadedFile:0x007f5363a7e730 @tempfile=#<Tempfile:/tmp/RackMultipart20170809-23802-vkohnm.rpm>, @original_filename="gofer-2.7.6-1.el7sat.noarch.rpm", @content_type="application/x-rpm", @headers="Content-Disposition: form-data; name=\"content[]\"; filename=\"gofer-2.7.6-1.el7sat.noarch.rpm\"\r\nContent-Type: application/x-rpm\r\n">, #<ActionDispatch::Http::UploadedFile:0x007f5363a7e708 @tempfile=#<Tempfile:/tmp/RackMultipart20170809-23802-17h42oj.rpm>, @original_filename="hiera-1.3.1-2.el7sat.noarch.rpm", @content_type="application/x-rpm", @headers="Content-Disposition: form-data; name=\"content[]\"; filename=\"hiera-1.3.1-2.el7sat.noarch.rpm\"\r\nContent-Type: application/x-rpm\r\n">, #<ActionDispatch::Http::UploadedFile:0x007f5363a7e6e0 @tempfile=#<Tempfile:/tmp/RackMultipart20170809-23802-otdn97.rpm>, @original_filename="katello-agent-2.5.0-5.el7sat.noarch.rpm", @content_type="application/x-rpm", @headers="Content-Disposition: form-data; name=\"content[]\"; filename=\"katello-agent-2.5.0-5.el7sat.noarch.rpm\"\r\nContent-Type: application/x-rpm\r\n">, #<ActionDispatch::Http::UploadedFile:0x007f5363a7e6b8 @tempfile=#<Tempfile:/tmp/RackMultipart20170809-23802-v0xw3u.rpm>, @original_filename="katello-agent-2.9.0-2.el7sat.noarch.rpm", @content_type="application/x-rpm", @headers="Content-Disposition: form-data; name=\"content[]\"; filename=\"katello-agent-2.9.0-2.el7sat.noarch.rpm\"\r\nContent-Type: application/x-rpm\r\n">, #<ActionDispatch::Http::UploadedFile:0x007f5363a7e690 @tempfile=#<Tempfile:/tmp/RackMultipart20170809-23802-gftxtw.rpm>, @original_filename="libtasn1-3.8-3.el7.x86_64.rpm", @content_type="application/x-rpm", @headers="Content-Disposition: form-data; name=\"content[]\"; filename=\"libtasn1-3.8-3.el7.x86_64.rpm\"\r\nContent-Type: application/x-rpm\r\n">, #<ActionDispatch::Http::UploadedFile:0x007f5363a7e668 @tempfile=#<Tempfile:/tmp/RackMultipart20170809-23802-n3gpe8.rpm>, @original_filename="pulp-puppet-tools-2.8.7.1-1.el7sat.noarch.rpm", @content_type="application/x-rpm", @headers="Content-Disposition: form-data; name=\"content[]\"; filename=\"pulp-puppet-tools-2.8.7.1-1.el7sat.noarch.rpm\"\r\nContent-Type: application/x-rpm\r\n">, #<ActionDispatch::Http::UploadedFile:0x007f5363a7e640 @tempfile=#<Tempfile:/tmp/RackMultipart20170809-23802-1y4hy5r.rpm>, @original_filename="pulp-rpm-handlers-2.8.7.5-1.el7sat.noarch.rpm", @content_type="application/x-rpm", @headers="Content-Disposition: form-data; name=\"content[]\"; filename=\"pulp-rpm-handlers-2.8.7.5-1.el7sat.noarch.rpm\"\r\nContent-Type: application/x-rpm\r\n">, #<ActionDispatch::Http::UploadedFile:0x007f5363a7e618 @tempfile=#<Tempfile:/tmp/RackMultipart20170809-23802-xvx3xz.rpm>, @original_filename="puppet-3.8.6-2.el7sat.noarch.rpm", @content_type="application/x-rpm", @headers="Content-Disposition: form-data; name=\"content[]\"; filename=\"puppet-3.8.6-2.el7sat.noarch.rpm\"\r\nContent-Type: application/x-rpm\r\n">, #<ActionDispatch::Http::UploadedFile:0x007f5363a7e5f0 @tempfile=#<Tempfile:/tmp/RackMultipart20170809-23802-1d75hnv.rpm>, @original_filename="python-gofer-2.7.6-1.el7sat.noarch.rpm", @content_type="application/x-rpm", @headers="Content-Disposition: form-data; name=\"content[]\"; filename=\"python-gofer-2.7.6-1.el7sat.noarch.rpm\"\r\nContent-Type: application/x-rpm\r\n">, #<ActionDispatch::Http::UploadedFile:0x007f5363a7e5c8 @tempfile=#<Tempfile:/tmp/RackMultipart20170809-23802-xhy32d.rpm>, @original_filename="python-gofer-proton-2.7.6-1.el7sat.noarch.rpm", @content_type="application/x-rpm", @headers="Content-Disposition: form-data; name=\"content[]\"; filename=\"python-gofer-proton-2.7.6-1.el7sat.noarch.rpm\"\r\nContent-Type: application/x-rpm\r\n">], "authenticity_token"=>"9XVAn2T4kyTv3Yx26C2WarrZN1A8gdCZP3xlhnpS9pgNgj4CWa9RXI/XOVn4CZKnz0MNlC5yDC+NZMqum2L0nA==", "api_version"=>"v2", "id"=>"20"}
2017-08-09 15:41:25 394054ad [app] [I] Current user: admin (administrator)
2017-08-09 15:41:29 394054ad [app] [I] Completed 200 OK in 4472ms (Views: 0.4ms | ActiveRecord: 10.4ms)

Comment 35 jcallaha 2017-08-09 20:11:07 UTC
Created attachment 1311389 [details]
verification screenshot 1

Comment 36 jcallaha 2017-08-09 20:11:32 UTC
Created attachment 1311390 [details]
verification screenshot 2

Comment 37 Tanya Tereshchenko 2017-08-15 18:26:36 UTC
I'm associating another Pulp issue with this BZ. It's a fix for regression (same error but in sync) introduced with associated Pulp issue 1903.

Comment 38 pulp-infra@redhat.com 2017-08-15 18:32:28 UTC
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Comment 39 pulp-infra@redhat.com 2017-08-15 18:32:33 UTC
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.

Comment 44 jcallaha 2017-08-28 15:47:45 UTC
Verified in Satellite 6.2.12 Snap 1. 

I was able to successfully upload multiple rpms including one that has non-utf-8 characters in the header. See this for rpm specifics or the rpm I will attach.

https://github.com/PulpQE/pulp-fixtures/blob/master/rpm/assets-specs/rpm-with-non-utf-8.spec#L20-L21

Comment 45 jcallaha 2017-08-28 15:49:13 UTC
Created attachment 1319138 [details]
non-utf8 rpm

Comment 47 errata-xmlrpc 2017-09-25 18:59:44 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2803


Note You need to log in before you can comment on or make changes to this bug.