Description of problem: Synchronization of RHEL content for an Organization with an i18n name fails (seems to be at the end of the process) with the following error: UnicodeDecodeError: 'utf8' codec can't decode byte 0xe9 in position 270: invalid continuation byte Version-Release number of selected component (if applicable): * candlepin-0.6.5-1.el6_2.noarch * candlepin-tomcat6-0.6.5-1.el6_2.noarch * katello-0.1.318-1.el6cf.noarch * katello-all-0.1.318-1.el6cf.noarch * katello-candlepin-cert-key-pair-1.0-1.noarch * katello-certs-tools-1.0.7-1.el6_3.noarch * katello-cli-0.1.112-1.el6cf.noarch * katello-cli-common-0.1.112-1.el6cf.noarch * katello-common-0.1.318-1.el6cf.noarch * katello-configure-0.1.110-1.el6_3.noarch * katello-glue-candlepin-0.1.318-1.el6cf.noarch * katello-glue-foreman-0.1.318-1.el6cf.noarch * katello-glue-pulp-0.1.318-1.el6cf.noarch * katello-qpid-broker-key-pair-1.0-1.noarch * katello-qpid-client-key-pair-1.0-1.noarch * katello-selinux-0.1.10-1.el6.noarch * pulp-1.0.4-1.el6.noarch * pulp-common-1.0.4-1.el6.noarch * pulp-selinux-server-1.0.4-1.el6.noarch How reproducible: Steps to Reproduce: 1. Create user Mané with the Administrator role. Log in as this user. 2. Create organization: Cuiabá 3. Add the following environments: Library -> Desenvolvimento -> ControleQualidade 4. Upload a valid manifest file and enable the following repositories: RHEL 6Server RPMS, CF Tools and SAM (all x86_64) 5. Synchronize all of the enabled repositories Actual results: The sync process eventually breaks and the error already mentioned is found in pulp.log Expected results: Sync should be completed Additional info:
pulp.log: ========= 2012-06-25 20:58:27,563 17966:140086299576064: pulp.server.webservices.controllers.repositories:INFO: repositories:1175 sync timeout passed : None 2012-06-25 20:58:27,791 17966:140086692529920: pulp:CRITICAL: queue:130 Exception in FIFO Queue Dispatch Thread Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/pulp/server/tasking/taskqueue/queue.py", line 124, in _dispatch self.run(task) File "/usr/lib/python2.6/site-packages/pulp/server/tasking/taskqueue/queue.py", line 314, in run self.__storage.store_running(task) File "/usr/lib/python2.6/site-packages/pulp/server/tasking/taskqueue/storage.py", line 232, in store_running self._snapshot_task(task) File "/usr/lib/python2.6/site-packages/pulp/server/tasking/taskqueue/storage.py", line 223, in _snapshot_task snapshot = task.snapshot() File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync_task.py", line 64, in snapshot snapshot = super(RepoSyncTask, self).snapshot() File "/usr/lib/python2.6/site-packages/pulp/server/tasking/task.py", line 314, in snapshot snapshot = model.TaskSnapshot(data) File "/usr/lib/python2.6/site-packages/pulp/server/db/model/persistence.py", line 45, in __init__ self.update(self._process_serialized_task(serialized_task)) File "/usr/lib/python2.6/site-packages/pulp/server/db/model/persistence.py", line 58, in _process_serialized_task return dict([(k, _process_value(v)) for k, v in serialized_task.items()]) File "/usr/lib/python2.6/site-packages/pulp/server/db/model/persistence.py", line 55, in _process_value value = value.decode('utf-8') File "/usr/lib64/python2.6/encodings/utf_8.py", line 16, in decode return codecs.utf_8_decode(input, errors, True) UnicodeDecodeError: 'utf8' codec can't decode byte 0xe9 in position 270: invalid continuation byte 2012-06-25 20:58:27,797 17966:140086154868480: pulp.server.webservices.controllers.repositories:INFO: repositories:1175 sync timeout passed : None 2012-06-25 20:58:27,810 17966:140086692529920: pulp:CRITICAL: queue:130 Exception in FIFO Queue Dispatch Thread Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/pulp/server/tasking/taskqueue/queue.py", line 124, in _dispatch self.run(task) File "/usr/lib/python2.6/site-packages/pulp/server/tasking/taskqueue/queue.py", line 314, in run self.__storage.store_running(task) File "/usr/lib/python2.6/site-packages/pulp/server/tasking/taskqueue/storage.py", line 232, in store_running self._snapshot_task(task) File "/usr/lib/python2.6/site-packages/pulp/server/tasking/taskqueue/storage.py", line 223, in _snapshot_task snapshot = task.snapshot() File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync_task.py", line 64, in snapshot snapshot = super(RepoSyncTask, self).snapshot() File "/usr/lib/python2.6/site-packages/pulp/server/tasking/task.py", line 314, in snapshot snapshot = model.TaskSnapshot(data) File "/usr/lib/python2.6/site-packages/pulp/server/db/model/persistence.py", line 45, in __init__ self.update(self._process_serialized_task(serialized_task)) File "/usr/lib/python2.6/site-packages/pulp/server/db/model/persistence.py", line 58, in _process_serialized_task return dict([(k, _process_value(v)) for k, v in serialized_task.items()]) File "/usr/lib/python2.6/site-packages/pulp/server/db/model/persistence.py", line 55, in _process_value value = value.decode('utf-8') File "/usr/lib64/python2.6/encodings/utf_8.py", line 16, in decode return codecs.utf_8_decode(input, errors, True) UnicodeDecodeError: 'utf8' codec can't decode byte 0xe9 in position 270: invalid continuation byte 2012-06-25 20:58:27,953 17966:140086299576064: pulp.server.webservices.controllers.repositories:INFO: repositories:1175 sync timeout passed : None
Worth to observe that https://bugzilla.redhat.com/show_bug.cgi?id=817914 would prevent me from creating the organization but that is only available in pulp 1.1 release (current cfse has pulp-1.0.4-1.el6.noarch).
2012-06-26 10:32:13 < omaciel> jsherrill, jmatthews preethi filed https://bugzilla.redhat.com/show_bug.cgi?id=835586 I wanted to clarify something: Pulp 1.1 will disallow multi-byte repo names but is Katello aware of this? Will the error bubble up back up to Katello web ui? 2012-06-26 10:32:38 < omaciel> see the Bz mentioned on comment #2 2012-06-26 10:34:28 < jmatthews> omaciel, I am unsure about how Katello handles this, yet since Pulp is throwing an exception I'd think Katello will see the validation failed and display an error. 2012-06-26 10:35:31 < omaciel> jmatthews: that's why I included jsherrill in the question... want to make sure we can handle it... 2012-06-26 10:36:08 < omaciel> jmatthews: now, will something like mánáña be acceptable for a repo name in pulp 1.1? 2012-06-26 10:37:43 < jmatthews> omaciel, I don't think so 2012-06-26 10:38:10 < mhrivnak> jdob: I do the same. I like to make incremental commits, sometimes just for convenient workflow reasons, and then do a squash merge at the end so I can make "official" commits. That helps me best organize commits to have reasonable scope and commit messages, so they make sense to others and match the assigned tasks. 2012-06-26 10:38:40 < jsherrill> omaciel: well, we should bubble it backup, but since org and environment name are used to construct the repo name, you may not see the error till its too late and you've already created those things (such as your bug) 2012-06-26 10:38:50 < jsherrill> so we really need to catch this within katello 2012-06-26 10:38:51 < jdob> exactly, and whenver I do something really awesome (which, let's be honest, is most things I do), I worry my hard drive is gonna explode and I'll lose it, so i like to back up 2012-06-26 10:39:43 < omaciel> jsherrill: seems that the web ui then should disallow the type of org name I chose, which to me imvho is not a good thing 2012-06-26 10:40:02 < omaciel> basically we're saying that people cannot create orgs with i18n names 2012-06-26 10:40:05 < jsherrill> omaciel: you mean its not a good thing to dissallow org names with i18n ? 2012-06-26 10:40:10 < omaciel> correct 2012-06-26 10:40:14 < jsherrill> i would agree 2012-06-26 10:40:27 < omaciel> so that puts us in a tough place 2012-06-26 10:40:27 < jsherrill> but unless we take out org and environment name from the repo name, not much we can do 2012-06-26 10:40:51 < jsherrill> and we could take those out, it would just make it harder to identify repos 2012-06-26 10:40:56 < jsherrill> from the client side 2012-06-26 10:41:01 < jsherrill> (important) 2012-06-26 10:41:07 < jsherrill> and the pulp side for debugging (less important) 2012-06-26 10:41:15 < omaciel> jsherrill: who can make this decision? 2012-06-26 10:41:28 < jsherrill> not sure honestly :)
I have the feeling that the issue here is caused only when the user initializing the sync process has a username with latin-1 characters (e.g. Mané)
Ok, checked both 1.0.0 and 1.0.1 releases and the issue does indeed seem to be related to the latin-1 username performing the sync process. One thing that is also visible is the lack of a progress bar in the web ui.
Here's how to reproduce it: alias kk='/usr/bin/katello -u admin -p admin' kk user create --username "Mané" --password redhat --email "mane" kk user assign_role --username "Mané" --role "Administrator" alias kk='/usr/bin/katello -u Mané -p redhat' kk org create --name Brasília kk environment create --org Brasília --name Dev1 --prior Library kk client remember --option org --value Brasília kk provider create --org "Brasília" --name Brew kk product create --org "Brasília" --provider Brew --name "SE" kk changeset create --org "Brasília" --name "Promote to Dev1" --environment Dev1 kk repo create --org "Brasília" --product "SE" --name "SE x86_64" --url http://download.devel.redhat.com/brewroot/repos/se-rhel-6-build/latest/x86_64/ kk provider synchronize --org "Brasília" --name Brew This last command shows: undefined method `[]' for nil:NilClass
Possibly related: User cannot apparently delete an org with latin-1 name, possibly if it contains pulp content. While assisting with this bug, I created an org, "Cuibá", created some third party repos and synced. Attempting to delete, I get a delete failed notification: Failed to delete organization 'Cuibá'. Check notices for more details. bad URI(is not URI?): http://localhost:9200/katello_package/_query?q=repoids:Cuibá-x86_64-google_chrome
ACK for 1.1, we will encode the names and IDs in a way that works with Pulp
This has nothing to do with the data. The error is in code itself. In /usr/lib/python2.6/site-packages/pulp/server/db/model/persistence.py there is in _process_serialized_task() if value is not unicode: value = value.decode('utf-8') and see this: >>> value=u'Mané' >>> value is not unicode True >>> value.decode('utf-8') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib64/python2.7/encodings/utf_8.py", line 16, in decode return codecs.utf_8_decode(input, errors, True) UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 3: ordinal not in range(128) there shoudl be obviously: if not (isinstance(value, unicode): but since isinstace() is quite slow procedure, you can replace it by: if not (type(value) == unicode): if you are absolutly sure you do not have child class from unicode.
the above code is indeed a bug but we updated to using ASCII labels for repo IDs so the above code shouldn't have an issue. Will test with our fixes in place
The code which cause the traceback in #1 and which I pointed in #9 is still badly written. Either remove the condition (as it is always true in current code), or even better fix it as I suggested - so the condition will be true only if type of value is unicode. Additionally somebody (not strictly katello) will give unicode string with accents on input, it should raise some exception or give friendly error. But definitelly not fail with traceback.
These should be fixed labels changes where we are forcing the user to use ascii labels to sync, seems to work ok after the changes specified in https://github.com/Katello/katello/pull/724
[root@qetello02 ~]# alias kk='/usr/bin/katello -u admin -p admin' [root@qetello02 ~]# kk user create --username "Mané" --password redhat --email "mane" Successfully created user [ Mané ] [root@qetello02 ~]# kk user assign_role --username "Mané" --role "Administrator" User 'Mané' assigned to role 'Administrator' [root@qetello02 ~]# alias kk='/usr/bin/katello -u Mané -p redhat' [root@qetello02 ~]# kk org create --name Brasília Successfully created org [ Brasília ] [root@qetello02 ~]# kk environment create --org Brasília --name Dev1 --prior Library Couldn't find organization 'Brasília' [root@qetello02 ~]# kk environment create --org 2c0049c3-d57a-4039-8833-9697fa850c9b --name Dev1 --prior Library Successfully created environment [ Dev1 ] [root@qetello02 ~]# kk client remember --option org --value 2c0049c3-d57a-4039-8833-9697fa850c9b Successfully remembered option [ org ] [root@qetello02 ~]# kk provider create --name Brew Successfully created provider [ Brew ] [root@qetello02 ~]# kk product create --provider Brew --name "SE" Successfully created product [ SE ] [root@qetello02 ~]# kk changeset create --name "Promote to Dev1" --environment Dev1 Successfully created changeset [ Promote to Dev1 ] for environment [ Dev1 ] [root@qetello02 ~]# kk repo create --product "SE" --name "SE x86_64" --url http://download.devel.redhat.com/brewroot/repos/se-rhel-6-build/latest/x86_64/ Successfully created repository [ SE x86_64 ] [root@qetello02 ~]# kk provider synchronize --name Brew undefined method `[]' for nil:NilClass
Could not be verified even though 803702 is yet to be fixed. I tried to use the value for Label as Org (see my command history above) and sync did not work. * candlepin-0.7.8-1.el6cf.noarch * candlepin-selinux-0.7.8-1.el6cf.noarch * candlepin-tomcat6-0.7.8-1.el6cf.noarch * katello-1.1.12-9.el6cf.noarch * katello-all-1.1.12-9.el6cf.noarch * katello-candlepin-cert-key-pair-1.0-1.noarch * katello-certs-tools-1.1.8-1.el6cf.noarch * katello-cli-1.1.8-5.el6cf.noarch * katello-cli-common-1.1.8-5.el6cf.noarch * katello-common-1.1.12-9.el6cf.noarch * katello-configure-1.1.9-4.el6cf.noarch * katello-glue-candlepin-1.1.12-9.el6cf.noarch * katello-glue-pulp-1.1.12-9.el6cf.noarch * katello-qpid-broker-key-pair-1.0-1.noarch * katello-qpid-client-key-pair-1.0-1.noarch * katello-selinux-1.1.1-1.el6cf.noarch * pulp-1.1.12-1.el6cf.noarch * pulp-common-1.1.12-1.el6cf.noarch * pulp-selinux-server-1.1.12-1.el6cf.noarch
associated pulp bug: https://bugzilla.redhat.com/show_bug.cgi?id=862068
oops, wrong BZ, meant: https://bugzilla.redhat.com/show_bug.cgi?id=862356
*** Bug 862068 has been marked as a duplicate of this bug. ***
build: 1.1.13
seems like this is still not fixed in pulp build 1.1.13 [root@pulp-v1-server ~]# pulp-admin user create --username Månə --password redhat Successfully created user [ Månə ] with name [ None ] [root@pulp-v1-server ~]# [root@pulp-v1-server ~]# pulp-admin user list +------------------------------------------+ Available Users +------------------------------------------+ Login : admin Name : None Roles : super-users Login : Månə Name : None Roles : [root@pulp-v1-server ~]# pulp-admin permission grant --resource=/ --user Månə -o create -o update -o delete Operations ['CREATE', 'UPDATE', 'DELETE'] granted to user [ Månə ] on resource [ / ] [root@pulp-v1-server ~]# pulp-admin permission show --resource / +------------------------------------------+ Permissions for / +------------------------------------------+ admin CREATE, READ, UPDATE, DELETE, EXECUTE Månə CREATE, UPDATE, DELETE [root@pulp-v1-server ~]# pulp-admin permission grant --resource=/ --user Månə -o read -o execute Operations ['READ', 'EXECUTE'] granted to user [ Månə ] on resource [ / ] [root@pulp-v1-server ~]# pulp-admin -u Månə -p redhat repo create --id pulp --feed http://repos.fedorapeople.org/repos/pulp/pulp/v2/candidate/fedora-17/x86_64/ Successfully created repository [ pulp ] [root@pulp-v1-server ~]# [root@pulp-v1-server ~]# pulp.log [root@pulp-v1-server ~]# pulp-admin -u Månə -p redhat repo sync --id pulp -F Sync for repository pulp started Waiting / Task not found: 5e51ce80-0e50-11e2-a4e0-52540067ae07 Traceback (most recent call last): File "/usr/bin/pulp-admin", line 23, in <module> admin.main() File "/usr/lib/python2.6/site-packages/pulp/client/lib/cli.py", line 152, in main command.main(self.args[1:]) File "/usr/lib/python2.6/site-packages/pulp/client/pluginlib/command.py", line 129, in main action.main(args[1:]) File "/usr/lib/python2.6/site-packages/pulp/client/pluginlib/command.py", line 213, in main self.run() File "/usr/lib/python2.6/site-packages/pulp/client/admin/plugins/repo.py", line 823, in run self._foreground_final_output(final_task) File "/usr/lib/python2.6/site-packages/pulp/client/admin/plugins/repo.py", line 977, in _foreground_final_output state = task['state'] TypeError: 'NoneType' object is unsubscriptable
from pulp.log 2012-10-04 16:45:02,505 30144:140631915620096: pulp.server.webservices.controllers.repositories:INFO: repositories:1179 sync timeout passed : None 2012-10-04 16:45:02,588 30144:140632194545408: pulp:CRITICAL: queue:130 Exception in FIFO Queue Dispatch Thread Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/pulp/server/tasking/taskqueue/queue.py", line 124, in _dispatch self.run(task) File "/usr/lib/python2.6/site-packages/pulp/server/tasking/taskqueue/queue.py", line 313, in run self.__storage.store_running(task) File "/usr/lib/python2.6/site-packages/pulp/server/tasking/taskqueue/storage.py", line 232, in store_running self._snapshot_task(task) File "/usr/lib/python2.6/site-packages/pulp/server/tasking/taskqueue/storage.py", line 223, in _snapshot_task snapshot = task.snapshot() File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync_task.py", line 64, in snapshot snapshot = super(RepoSyncTask, self).snapshot() File "/usr/lib/python2.6/site-packages/pulp/server/tasking/task.py", line 314, in snapshot snapshot = model.TaskSnapshot(data) File "/usr/lib/python2.6/site-packages/pulp/server/db/model/persistence.py", line 45, in __init__ self.update(self._process_serialized_task(serialized_task)) File "/usr/lib/python2.6/site-packages/pulp/server/db/model/persistence.py", line 58, in _process_serialized_task return dict([(k, _process_value(v)) for k, v in serialized_task.items()]) File "/usr/lib/python2.6/site-packages/pulp/server/db/model/persistence.py", line 55, in _process_value value = value.decode('utf-8') File "/usr/lib64/python2.6/encodings/utf_8.py", line 16, in decode return codecs.utf_8_decode(input, errors, True) UnicodeDecodeError: 'utf8' codec can't decode byte 0xe5 in position 268: invalid continuation byte
@Mike: is it true that we don't really need a new version of Pulp but a fix that you were working on to change the headers for the request being made?
Yes, PR: https://github.com/Katello/katello/pull/799
No new pulp version needed for now, they fixed the above code mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=835586#c9 but that will be part of V2
Merged into katello master: Branch: refs/heads/master Commit: e71ff05458ac9e9155d77f7f96848cc502bcfd3e
the latest rev of pulp still does not properly decode. looking into why
Waiting for pulp fix.
Narrowed it down a bit more into a reproducer bash script found here: https://bugzilla.redhat.com/show_bug.cgi?id=867647
The final resolution to this bug will be the requirement that we restrict usernames in Katello to being ASCII only. Pulp is unable to support login/usernames with characters that are beyond the ASCII charset so we will be adding field validation to the username field in Katello to restrict it as such. Docs will need to be updated to reflect this: http://documentation-devel.engineering.redhat.com/docs/en-US/CloudForms/1.1/html/Release_Notes/test11.html the above rel-note mentions usernames which we need to take off.
https://github.com/Katello/katello/pull/907
qa -> cwelton
Marking this Verified. Per changes above (and tested), usernames cannot at this point be utf8. This implementation means we no longer see the behavior listed in this bug. Repeated these steps using a non-utf8 username and things seem to work ok. CloudForms System Engine Version: 1.1.12-18.el6cf
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHSA-2012-1543.html