Bug 2079946
| Summary: | uploading package with changelog with non-UTF8 character fails | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Satellite | Reporter: | Pavel Moravec <pmoravec> | ||||
| Component: | Pulp | Assignee: | satellite6-bugs <satellite6-bugs> | ||||
| Status: | CLOSED WONTFIX | QA Contact: | Lai <ltran> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | medium | ||||||
| Version: | 6.10.4 | CC: | dalley, dkliban, ggainey, momran, rbdiri, rchan | ||||
| Target Milestone: | Unspecified | Keywords: | Triaged | ||||
| Target Release: | Unused | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2022-10-06 12:30:47 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
Created attachment 1875707 [details]
example RPM with nonUTF8 chars in changelog
See this thread for discussion on how Pulp 2 historically handled this (replacement characters): https://pulp.plan.io/issues/1903 The best Pulp could really do here is to strip out (or substitute) the non-UTF8 data so that it can't cause harm. But that is of course modifying the data after the fact, so that Pulp will not exactly match the package. The most ideal solution is therefore to not use non-UTF8 characters in the specfile, so that they do not end up in the resulting packages. Both Fedora and SUSE Packaging guidelines require this: https://docs.fedoraproject.org/en-US/packaging-guidelines/#_spec_file_encoding https://en.opensuse.org/openSUSE:Specfile_guidelines#Specfile_Encoding It seems that modern versions of rpmbuild also try to enforce this unless you explicitly disable the check: https://github.com/rpm-software-management/rpm/issues/952#issuecomment-558618524= Since the ecosystem has moved towards utf-8 to such a degree that this is the first and only report of the problem that I'm aware of, I'm inclined towards enforcing it as well unless we can find more examples. The error message, however, should probably be more friendly. I agree there is no reason pulp should support something beyond packaging guidelines, and that packages with such non-UTF8 character data are invalid input for pulp/Satellite. It makes sense to improve the error message to have there something more understandable / pointing to the cause of problems. Since this issue hadn't been reported until recently and there is a KCS (linked) explaining the error, I see rather smaller (though still legitimate) benefit from this improvement. Closing this bugzilla as per confirmation by the customer that the workaround documented in the following KB solution works for them: Uploading a file on custom repo it fails with error : Katello::Errors::Pulp3Error: <built-in function getattr> returned a result with an error set Link: https://access.redhat.com/solutions/6956710 |
Description of problem: Having a changelog with non-UTF-8 characters (like Czech or French specific letters from alphabets), such RPM fails to be uploaded to a pulp/Satellite repo. Version-Release number of selected component (if applicable): Sat 6.10.4 python3-pulpcore-3.14.13-1.el7pc.noarch python3-pulp-rpm-3.14.12-1.el7pc.noarch How reproducible: 100% Steps to Reproduce: 1. Upload a package with such changelog to Satellite repo. Example of such a package will be appended. 2. Optionally, test the critical piece of code from pulpcore-manager shell (after uploading the file to /tmp dir): # cat pulp-unicode-error.py from pulp_rpm.app.models.package import Package from django.conf import settings import createrepo_c cr_pkginfo = createrepo_c.package_from_rpm('/tmp/pulp-unicode-1-0.el8.x86_64.rpm', changelog_limit=settings.KEEP_CHANGELOG_LIMIT) print("======== cr_pkginfo.changelogs =========") print(cr_pkginfo.changelogs) print("======== createrepo_to_dict: ========") package = Package.createrepo_to_dict(cr_pkginfo) # cat pulp-unicode-error.py | sudo -u pulp PULP_SETTINGS='/etc/pulp/settings.py' DJANGO_SETTINGS_MODULE='pulpcore.app.settings' pulpcore-manager shell Actual results: 1. fails with error: Error during upload: Task 7a5b6e8d-7292-4d61-bb11-12263986d370: Katello::Errors::Pulp3Error: <built-in function getattr> returned a result with an error set 2. fails with uncaught exception: ======== cr_pkginfo.changelogs ========= [('Pavel Moravec <pmoravec> 1.0', 1651147200, <NULL>)]UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbf in position 3: invalid start byte The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/bin/pulpcore-manager", line 11, in <module> load_entry_point('pulpcore==3.14.13', 'console_scripts', 'pulpcore-manager')() File "/usr/lib/python3.6/site-packages/pulpcore/app/manage.py", line 11, in manage execute_from_command_line(sys.argv) File "/usr/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line utility.execute() File "/usr/lib/python3.6/site-packages/django/core/management/__init__.py", line 375, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/usr/lib/python3.6/site-packages/django/core/management/base.py", line 323, in run_from_argv self.execute(*args, **cmd_options) File "/usr/lib/python3.6/site-packages/django/core/management/base.py", line 364, in execute output = self.handle(*args, **options) File "/usr/lib/python3.6/site-packages/django/core/management/commands/shell.py", line 92, in handle exec(sys.stdin.read()) File "<string>", line 8, in <module> SystemError: <built-in method write of _io.TextIOWrapper object at 0x7f5b83f80630> returned a result with an error set (or the "Package.createrepo_to_dict(cr_pkginfo)" fails with : >>> package = Package.createrepo_to_dict(cr_pkginfo) UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbf in position 3: invalid start byte The above exception was the direct cause of the following exception: Traceback (most recent call last): File "<console>", line 1, in <module> File "/usr/lib/python3.6/site-packages/pulp_rpm/app/models/package.py", line 278, in createrepo_to_dict SystemError: <built-in function getattr> returned a result with an error set ) Expected results: Upload succeeds, no errors in pulpcore-manager. Additional info: