Bug 1474872 - rhncfg-manager download-channel failed during download utf8
rhncfg-manager download-channel failed during download utf8
Status: CLOSED CURRENTRELEASE
Product: Spacewalk
Classification: Community
Component: Clients (Show other bugs)
2.6
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Eric Herget
Red Hat Satellite QA List
: Reopened
Depends On:
Blocks: sat58-errata space28
  Show dependency treegraph
 
Reported: 2017-07-25 10:29 EDT by Pavel Studeník
Modified: 2018-04-20 08:31 EDT (History)
2 users (show)

See Also:
Fixed In Version: rhncfg-5.10.115-1
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2018-04-20 08:31:36 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Pavel Studeník 2017-07-25 10:29:20 EDT
Description of problem:
I created configure channel with file that contains utf-8 characters. For example +ěščřžýáíé. When I try to download and backup the configure channel I receive following error:

>> rhncfg-manager download-channel .. --topdir=..
...
UnicodeEncodeError: 'ascii' codec can't encode characters in position 38-46: ordinal not in range(128)

Version-Release number of selected component (if applicable):
rhncfg-5.10.105-1.fc25.noarch

How reproducible:
always

Steps to Reproduce:
1. create configure channel 
2. add file "+ěščřž" to the configure channel
3. run: rhncfg-manager download-channel conchtest --topdir=/tmp/confchtest/

Additional info:
Traceback (most recent call last):
  File "/usr/bin/rhncfg-manager", line 46, in <module>
    sys.exit(Main().main() or 0)
  File "/usr/share/rhn/config_common/rhn_main.py", line 218, in main
    handler.run()
  File "/usr/share/rhn/config_management/rhncfg_download_channel.py", line 52, in run
    config_channel=ns)
  File "/usr/share/rhn/config_common/deploy.py", line 43, in deploy_files
    finfo = repository.get_file_info(*args, **kwargs)
  File "/usr/share/rhn/config_management/rpc_repository.py", line 116, in get_file_info
    fullpath, dirs_created = fp.process(result, directory=dest_directory, strict_ownership=0)
  File "/usr/share/rhn/config_common/file_utils.py", line 106, in process
    fh.write(sstr(contents))
UnicodeEncodeError: 'ascii' codec can't encode characters in position 38-46: ordinal not in range(128)
Comment 1 Eric Herget 2017-07-28 11:01:21 EDT
I have tested this with spacewalk-nightly on RHEL7 (both server and client) and the rhncfg-manager download-channel command worked as it is expected to work.  The file it downloaded contained the utf8 characters.

So it does not appear to be an issue with latest code on rhel7.

I will test on Fedora 25 as this should use python 3 and may be the reason for the issue.
Comment 2 Eric Herget 2017-07-28 13:12:09 EDT
I also tested on f25 server and client and again the utf8 chars downloaded as expected.

I verified the f25 rhncfg-manager uses python 3, vs. python 2 on rhel7.

Finally, I tested with rhel7 client registered with f25 server, and f25 client registered with rhel7.  Both of these also worked as expected.
Comment 3 Eric Herget 2017-07-28 13:17:23 EDT
Closing as the issue has been resolved since it was reported.  This will go out with the upcoming Spacewalk 2.7 release.
Comment 4 Patrik Segedy 2017-09-04 10:42:03 EDT
I am able to reproduce it with SW nightly on Fedora 25 when locale is set to C (it uses ascii) or whatever else what is not using utf-8.

Reproducer with rhncfg-5.10.110-1.fc25.noarch
1. Create configuration channel conf_channel
2. echo "+ľščťžý" > utf-8_file
3. rhncfg-manager add -c conf_channel utf-8_file
4. export LC_ALL=C
5. rhncfg-manager download-channel conf_channel --topdir=$(mktemp -d)
Using server name https://sw-nightly-f25.example.com/CONFIG-MANAGEMENT-TOOL
Password: 
Traceback (most recent call last):
  File "/usr/bin/rhncfg-manager", line 46, in <module>
    sys.exit(Main().main() or 0)
  File "/usr/share/rhn/config_common/rhn_main.py", line 218, in main
    handler.run()
  File "/usr/share/rhn/config_management/rhncfg_download_channel.py", line 52, in run
    config_channel=ns)
  File "/usr/share/rhn/config_common/deploy.py", line 43, in deploy_files
    finfo = repository.get_file_info(*args, **kwargs)
  File "/usr/share/rhn/config_management/rpc_repository.py", line 116, in get_file_info
    fullpath, dirs_created = fp.process(result, directory=dest_directory, strict_ownership=0)
  File "/usr/share/rhn/config_common/file_utils.py", line 106, in process
    fh.write(sstr(contents))
UnicodeEncodeError: 'ascii' codec can't encode characters in position 1-6: ordinal not in range(128)

There is same problem if you use some other non utf-8 locale, e.g. LC_ALL=en_US. Then you'll get:
UnicodeEncodeError: 'latin-1' codec can't encode characters in position 1-5: ordinal not in range(256)
Comment 5 Eric Herget 2017-10-12 13:58:58 EDT
This is now fixed.  The problem was that we were using the file object returned by os.fdopen() to write the config files and doing this involves some higher level things happening that includes string processing based on the user's environment settings.  I modified this to use the lower level write operations where the fd is passed to os.write(fd, ...), which avoids the higher level stuff.
Comment 6 Eric Herget 2017-10-12 13:59:42 EDT
This also fixes a problem with writing binary config files on a python3 system.
Comment 7 Eric Herget 2017-10-12 14:00:27 EDT
spacewalk.github:
c160e591ed8f7e182bd74242bfc9e82643afaf70
Comment 8 Jiří Dostál 2018-03-26 08:16:38 EDT
Moving ON_QA
Comment 9 Jiří Dostál 2018-04-20 08:31:36 EDT
Spacewalk 2.8 has been released.
https://github.com/spacewalkproject/spacewalk/wiki/ReleaseNotes28

Note You need to log in before you can comment on or make changes to this bug.