Bug 1012628

Summary:	content: uploading puppet module with multibyte in name/manifest throws python error
Product:	Red Hat Satellite	Reporter:	Corey Welton <cwelton>
Component:	Content Management	Assignee:	David Davis <daviddavis>
Status:	CLOSED NOTABUG	QA Contact:	Katello QA List <katello-qa-list>
Severity:	low	Docs Contact:
Priority:	unspecified
Version:	Nightly	CC:	ehelms
Target Milestone:	Unspecified	Keywords:	Triaged
Target Release:	Unused
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2013-10-15 15:29:23 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1016310
Bug Blocks:

Description Corey Welton 2013-09-26 18:16:41 UTC

Description of problem:


Version-Release number of selected component (if applicable):
1.4.6-13.el6sat

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:
1.  Attempt to upload a puppet file with multibyte characters in the name.  Note: if you are trying to modify existing module for testing purposes, you'll need to edit both the filename and the name in the manifest itself.

 repo content_upload --repo=zooup --product=Zooshop --filepath=./puppetfoo/你好-good-2.0.0.tar.gz --content_type=puppet --org=ACME_Corporation
2. View results

Current results:
/usr/lib64/python2.6/tarfile.py:2350: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  if name == members[i].name:
Invalid puppet module './puppetfoo/你好-good-2.0.0.tar.gz'. Please check the file and try again.
[root@qe-blade-05 ~]# kk --version


Expected results:
Either disallow multibyte filenames (if applicable) or handle error gracefully or handle properly.

Additional info:

Comment 2 David Davis 2013-10-07 21:03:03 UTC

While we need some fixes to support this, the error you hit is incorrect. I'm guessing the author in your metadata.json wasn't 你好. I've created a unicode puppet module and it looks like we're hitting an error in pulp:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 1: ordinal not in range(128)
2013-10-07 17:01:39,015 pulp.server.dispatch.task:ERROR: Pulp exception occurred: PulpExecutionException
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp/server/dispatch/task.py", line 138, in _run
    result = call(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/pulp/server/managers/content/upload.py", line 227, in import_uploaded_unit
    importer_instance.upload_unit(transfer_repo, unit_type_id, unit_key, unit_metadata, file_path, conduit, call_config)
  File "/usr/lib/python2.6/site-packages/pulp_puppet/plugins/importers/importer.py", line 75, in upload_unit
    upload.handle_uploaded_unit(repo, type_id, unit_key, metadata, file_path, conduit)
  File "/usr/lib/python2.6/site-packages/pulp_puppet/plugins/importers/upload.py", line 52, in handle_uploaded_unit
    metadata_parser.extract_metadata(module, file_path, repo.working_dir)
  File "/usr/lib/python2.6/site-packages/pulp_puppet/plugins/importers/metadata.py", line 81, in extract_metadata
    metadata_json = _extract_non_standard_json(module, filename, temp_dir)
  File "/usr/lib/python2.6/site-packages/pulp_puppet/plugins/importers/metadata.py", line 132, in _extract_non_standard_json
    extraction_dir = os.path.join(temp_dir, module.author, module.name, module.version)
  File "/usr/lib64/python2.6/posixpath.py", line 70, in join
    path += '/' + b
PulpExecutionException: Pulp exception occurred: PulpExecutionException
2013-10-07 17:01:39,016 pulp.server.dispatch.task:INFO: FAILURE: Task 8f735c0e-1551-42d7-baab-b5b3d8684cbc: CallRequest: ContentUploadManager.import_uploaded_unit(u'ACME_Corporation-PForge_1381163037581-pforge', 'puppet_module', {'version': '2.0.0', 'name': 'good', 'author': '\xe4\xbd\xa0\xe5\xa5\xbd'}, {'description': 'NTP Module for Debian, Ubuntu, CentOS, RHEL, OEL, Fedora, FreeBSD, ArchLinux and Gentoo.', 'license': 'Apache Version 2.0', 'author': 'Puppet Labs', 'project_page': 'http://github.com/puppetlabs/puppetlabs-ntp', 'summary': 'NTP Module', 'source': 'git://github.com/puppetlabs/puppetlabs-ntp', 'version': '2.0.0', 'dependencies': [{'name': 'puppetlabs/stdlib', 'version_requirement': '>= 0.1.6'}], 'checksums': {'manifests/config.pp': '8d9afb6e4327277c96c5617ad687043a', 'spec/system/basic_spec.rb': 'f0fa6706b1011be24759dc06ccbd91e2', 'Gemfile': '779e8b104b94622ac8c7d1d7bcf32c5a', 'LICENSE': 'f0b6fdc310531526f257378d7bad0044', 'tests/init.pp': 'd398e7687ec1d893ef23d1b7d2afc094', 'spec/system/restrict_spec.rb': 'eb9d8deb680830f78e0041bb3c7d2326', 'manifests/params.pp': '2fe94c599d794811d950bd48879439d7', 'manifests/init.pp': '8883844f75084d31b4e5a9a50c900d87', 'spec/system/ntp_config_spec.rb': 'd60bc7ba907c080d891245d6f13da307', '.bundle/config': '7f1c988748783d2a8d455376eed1470c', 'spec/system/ntp_install_spec.rb': '9b3796bb71472daa3a52aa01b0592656', 'spec/unit/puppet/type/README.markdown': 'de26a7643813abd6c2e7e28071b1ef94', '.fixtures.yml': '909729694bab62c1e36001512b68a8fd', 'spec/spec_helper.rb': '0db89c9a486df193c0e40095422e19dc', 'spec/system/class_spec.rb': '83dcabc0e04432b52939ae8a5df992ca', 'spec/system/ntp_service_spec.rb': 'fbeec08f5489ce8ef82f2225f7107b9e', 'templates/ntp.conf.erb': '262e0a58bcd88d0cb65369081db30b32', 'CONTRIBUTING.md': '2ef1d6f4417dde9af6c7f46f5c8a864b', 'spec/spec.opts': 'a600ded995d948e393fbe2320ba8e51c', 'spec/unit/puppet/provider/README.markdown': 'e52668944ee6af2fb5d5b9e798342645', 'CHANGELOG': '1ef0e3fa661bd1b8cb8e1233343dd81c', 'README.markdown': '74a9a19e7866998e5a594845b15b0e57', 'Gemfile.lock': '1abd1da36ca6957b6d456ddd75e2a665', 'Rakefile': '0428ea3759a4692c91604396c406a9c1', 'spec/system/preferred_servers_spec.rb': '379fe9b56c10903178f03d4d87002a14', 'spec/classes/ntp_spec.rb': 'ec9345d2bb9976c012b117b8c7f8f052', 'manifests/service.pp': '350238b50e9cb896d270a2c76a64334f', 'spec/fixtures/modules/my_ntp/templates/ntp.conf.erb': '566e373728e9b13eda516115ff0a9fb0', 'Modulefile': 'c806ff020129ab4a367a502be3f1be75', '.nodeset.yml': '8d1b7762d4125ce53379966a1daf355c', 'manifests/install.pp': 'ac33c5733f4321a9af7a4735265c1986', '.travis.yml': '193ec2b14cc9644c88f4249422580da2', 'spec/spec_helper_system.rb': '4c85f6e43892c45ce75ade44a32f45f8'}, 'types': None, 'name': '\xe4\xbd\xa0\xe5\xa5\xbd-good'}, 'ce87ab49-5126-4349-909c-d3d68d2f882a')
2013-10-07 17:01:39,057 pulp.server.webservices.middleware.exception:ERROR: Pulp exception occurred: PulpExecutionException
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/middleware/exception.py", line 44, in __call__
    return self.app(environ, start_response)
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/middleware/postponed.py", line 39, in __call__
    return self.app(environ, start_response)
  File "/usr/lib/python2.6/site-packages/web/application.py", line 279, in wsgi
    result = self.handle_with_processors()
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in _handle_with_processors
    return process(self.processors)
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 23, in process
    return p(lambda : process(processors))
  File "/usr/lib/python2.6/site-packages/web/application.py", line 566, in processor
    return handler()
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 23, in <lambda>
    return p(lambda : process(processors))
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 23, in process
    return p(lambda : process(processors))
  File "/usr/lib/python2.6/site-packages/web/application.py", line 581, in processor
    result = handler()
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 23, in <lambda>

I tried a bunch of different formats (unicode, utf, etc) but I hit that error.

Comment 3 David Davis 2013-10-15 15:29:23 UTC

I found this thanks to #puppet:

http://docs.puppetlabs.com/puppet/3/reference/modules_fundamentals.html#allowed-module-names

Closing as NOTABUG per Mike McCune.