Bug 1072580

This error actually begins as an encoding error. The importer tries to report the encoding error using the progress report and the unit name as a key, and then the above traceback happens. Here's the progress report:

Apr 22 08:08:25 lemonade.rdu.redhat.com pulp[22065]: pulp_puppet.common.sync_progress:INFO: {'modules': {'finished_count': 0, 'state': 'running', 'traceback': None, 'error_count': 1, 'error': 'None', 'execution_time': None, 'total_count': 13, 'error_message': None, 'individual_errors': {u'xhgui-0.1.0-zeleznypa': {'exception': "'utf8' codec can't decode byte 0xbd in position 13: invalid start byte", 'traceback': [('/home/rbarlow/devel/pulp_puppet/pulp_puppet_plugins/pulp_puppet/plugins/importers/forge.py', 264, '_do_import_modules', 'self._add_new_module(downloader, module)'), ('/home/rbarlow/devel/pulp_puppet/pulp_puppet_plugins/pulp_puppet/plugins/importers/forge.py', 311, '_add_new_module', 'metadata.extract_metadata(module, unit.storage_path, self.repo.working_dir)'), ('/home/rbarlow/devel/pulp_puppet/pulp_puppet_plugins/pulp_puppet/plugins/importers/metadata.py', 87, 'extract_metadata', 'module.update_from_json(metadata_json)'), ('/home/rbarlow/devel/pulp_puppet/pulp_puppet_common/pulp_puppet/common/model.py', 166, 'update_from_json', 'parsed = json.loads(metadata_json)'), ('/usr/lib64/python2.7/json/__init__.py', 338, 'loads', 'return _default_decoder.decode(s)'), ('/usr/lib64/python2.7/json/decoder.py', 365, 'decode', 'obj, end = self.raw_decode(s, idx=_w(s, 0).end())'), ('/usr/lib64/python2.7/json/decoder.py', 381, 'raw_decode', 'obj, end = self.scan_once(s, idx)')]}}}, 'metadata': {'query_finished_count': 1, 'state': 'success', 'error_message': None, 'error': 'None', 'execution_time': 1, 'query_total_count': 1, 'current_query': u'http://forge.puppetlabs.com/modules.json?q=apache', 'traceback': None}}


/home/rbarlow/devel/pulp_puppet/pulp_puppet_plugins/pulp_puppet/plugins/importers/forge.py, Line 264 in _do_import_modules:
	self._add_new_module(downloader, module)s
/home/rbarlow/devel/pulp_puppet/pulp_puppet_plugins/pulp_puppet/plugins/importers/forge.py, Line 311 in _add_new_module:
	metadata.extract_metadata(module, unit.storage_path, self.repo.working_dir)s
/home/rbarlow/devel/pulp_puppet/pulp_puppet_plugins/pulp_puppet/plugins/importers/metadata.py, Line 87 in extract_metadata:
	module.update_from_json(metadata_json)s
/home/rbarlow/devel/pulp_puppet/pulp_puppet_common/pulp_puppet/common/model.py, Line 166 in update_from_json:
	parsed = json.loads(metadata_json)s
/usr/lib64/python2.7/json/__init__.py, Line 338 in loads:
	return _default_decoder.decode(s)s
/usr/lib64/python2.7/json/decoder.py, Line 365 in decode:
	obj, end = self.raw_decode(s, idx=_w(s, 0).end())s
/usr/lib64/python2.7/json/decoder.py, Line 381 in raw_decode:
	obj, end = self.scan_once(s, idx)s
'utf8' codec can't decode byte 0xbd in position 13: invalid start byte

I've spoken with Puppet Labs and they told me that this unicode error is the fault of the module author and not the fault of Pulp, so we will not fix it.

For the purposes of this bug, I have changed the way the sync progress report encodes errors, so we will now communicate the encoding error to the user. It is up to the module authors to fix the encoding problems, as they should all be using UTF-8.

I did report the problem with the xhgui package upstream:

https://github.com/zeleznypa/puppet-xhgui/issues/4

https://github.com/pulp/pulp_puppet/pull/114

This is included in today's pulp-2.4.0-0.11.beta build.

verified

[root@pulp-24-server ~]#  pulp-admin puppet repo create --repo-id=forge --feed=http://forge.puppetlabs.com --query apache
Successfully created repository [forge]

[root@pulp-24-server ~]#  pulp-admin puppet repo sync run --repo-id=forge
+----------------------------------------------------------------------+
                    Synchronizing Repository [forge]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.


Downloading metadata...
[==================================================] 100%
Metadata Query: 1/1 items
... completed

Downloading new modules...
[==================================================] 100%
Module: 60/60 items
... completed

Could not import the following modules:

    xhgui-0.1.0: 'utf8' codec can't decode byte 0xbd in position 13: invalid
start byte


Task Succeeded



Publishing modules...
[==================================================] 100%
Module: 59/59 items
... completed

Generating repository metadata...
[-]
... completed

Publishing repository over HTTP...
... completed

Publishing repository over HTTPS...
... skipped

Task Succeeded


[root@pulp-24-server ~]#

This has been fixed in Pulp 2.4.0-1.