Description of problem: Trying to sync a repo with feed http://forge.puppetlabs.com fails when there is units that has period in the name. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. Create a puppet repo with --feed http://forge.puppetlabs.com 2.Run repo sync 3. Actual results: [root@cloud-qe-17 puppet]# pulp-admin puppet repo create --repo-id=forge --feed=http://forge.puppetlabs.com --query apache Successfully created repository [forge] [root@cloud-qe-17 puppet]# pulp-admin puppet repo sync run --repo-id=forge+----------------------------------------------------------------------+ Synchronizing Repository [forge] +----------------------------------------------------------------------+ This command may be exited via ctrl+c without affecting the request. Downloading metadata... [==================================================] 100% Metadata Query: 1/1 items ... completed Downloading new modules... [===================== ] 43% Module: 24/55 items Task Failed key 'xhgui-0.1.0-zeleznypa' must not contain '.' Expected results: Additional info:
This error actually begins as an encoding error. The importer tries to report the encoding error using the progress report and the unit name as a key, and then the above traceback happens. Here's the progress report: Apr 22 08:08:25 lemonade.rdu.redhat.com pulp[22065]: pulp_puppet.common.sync_progress:INFO: {'modules': {'finished_count': 0, 'state': 'running', 'traceback': None, 'error_count': 1, 'error': 'None', 'execution_time': None, 'total_count': 13, 'error_message': None, 'individual_errors': {u'xhgui-0.1.0-zeleznypa': {'exception': "'utf8' codec can't decode byte 0xbd in position 13: invalid start byte", 'traceback': [('/home/rbarlow/devel/pulp_puppet/pulp_puppet_plugins/pulp_puppet/plugins/importers/forge.py', 264, '_do_import_modules', 'self._add_new_module(downloader, module)'), ('/home/rbarlow/devel/pulp_puppet/pulp_puppet_plugins/pulp_puppet/plugins/importers/forge.py', 311, '_add_new_module', 'metadata.extract_metadata(module, unit.storage_path, self.repo.working_dir)'), ('/home/rbarlow/devel/pulp_puppet/pulp_puppet_plugins/pulp_puppet/plugins/importers/metadata.py', 87, 'extract_metadata', 'module.update_from_json(metadata_json)'), ('/home/rbarlow/devel/pulp_puppet/pulp_puppet_common/pulp_puppet/common/model.py', 166, 'update_from_json', 'parsed = json.loads(metadata_json)'), ('/usr/lib64/python2.7/json/__init__.py', 338, 'loads', 'return _default_decoder.decode(s)'), ('/usr/lib64/python2.7/json/decoder.py', 365, 'decode', 'obj, end = self.raw_decode(s, idx=_w(s, 0).end())'), ('/usr/lib64/python2.7/json/decoder.py', 381, 'raw_decode', 'obj, end = self.scan_once(s, idx)')]}}}, 'metadata': {'query_finished_count': 1, 'state': 'success', 'error_message': None, 'error': 'None', 'execution_time': 1, 'query_total_count': 1, 'current_query': u'http://forge.puppetlabs.com/modules.json?q=apache', 'traceback': None}} /home/rbarlow/devel/pulp_puppet/pulp_puppet_plugins/pulp_puppet/plugins/importers/forge.py, Line 264 in _do_import_modules: self._add_new_module(downloader, module)s /home/rbarlow/devel/pulp_puppet/pulp_puppet_plugins/pulp_puppet/plugins/importers/forge.py, Line 311 in _add_new_module: metadata.extract_metadata(module, unit.storage_path, self.repo.working_dir)s /home/rbarlow/devel/pulp_puppet/pulp_puppet_plugins/pulp_puppet/plugins/importers/metadata.py, Line 87 in extract_metadata: module.update_from_json(metadata_json)s /home/rbarlow/devel/pulp_puppet/pulp_puppet_common/pulp_puppet/common/model.py, Line 166 in update_from_json: parsed = json.loads(metadata_json)s /usr/lib64/python2.7/json/__init__.py, Line 338 in loads: return _default_decoder.decode(s)s /usr/lib64/python2.7/json/decoder.py, Line 365 in decode: obj, end = self.raw_decode(s, idx=_w(s, 0).end())s /usr/lib64/python2.7/json/decoder.py, Line 381 in raw_decode: obj, end = self.scan_once(s, idx)s 'utf8' codec can't decode byte 0xbd in position 13: invalid start byte
I've spoken with Puppet Labs and they told me that this unicode error is the fault of the module author and not the fault of Pulp, so we will not fix it. For the purposes of this bug, I have changed the way the sync progress report encodes errors, so we will now communicate the encoding error to the user. It is up to the module authors to fix the encoding problems, as they should all be using UTF-8. I did report the problem with the xhgui package upstream: https://github.com/zeleznypa/puppet-xhgui/issues/4
https://github.com/pulp/pulp_puppet/pull/114
This is included in today's pulp-2.4.0-0.11.beta build.
verified [root@pulp-24-server ~]# pulp-admin puppet repo create --repo-id=forge --feed=http://forge.puppetlabs.com --query apache Successfully created repository [forge] [root@pulp-24-server ~]# pulp-admin puppet repo sync run --repo-id=forge +----------------------------------------------------------------------+ Synchronizing Repository [forge] +----------------------------------------------------------------------+ This command may be exited via ctrl+c without affecting the request. Downloading metadata... [==================================================] 100% Metadata Query: 1/1 items ... completed Downloading new modules... [==================================================] 100% Module: 60/60 items ... completed Could not import the following modules: xhgui-0.1.0: 'utf8' codec can't decode byte 0xbd in position 13: invalid start byte Task Succeeded Publishing modules... [==================================================] 100% Module: 59/59 items ... completed Generating repository metadata... [-] ... completed Publishing repository over HTTP... ... completed Publishing repository over HTTPS... ... skipped Task Succeeded [root@pulp-24-server ~]#
This has been fixed in Pulp 2.4.0-1.