Bug 1130057

Summary: It looks like Apache is not configured to accept non-ASCII file names
Product: OpenShift Online Reporter: dmt.kir <dmt.kir>
Component: ImageAssignee: Maciej Szulik <maszulik>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 2.xCC: bparees, dmt.kir, jokerman, mmccomas, wzheng
Target Milestone: ---Keywords: UpcomingRelease
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-10-10 00:50:33 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description dmt.kir@gmail.com 2014-08-14 09:03:03 UTC
Description of problem:

Looks like uploading of non-ascii filenames with the Django storage system with the current apache settings on my gear (app name "rv") triggers UnicodeEncodeError exceptions when calling functions like os.path()

http://rv-tdf.rhcloud.com/gallery/ with DEBUG=Tue =>
[...]
'ascii' codec can't encode character u'\u0301' in position 113: ordinal not in range(128)
[...]
The string that could not be encoded/decoded was: ils/Ávila,
[...]

PS.
(virtenv)[rv-tdf.rhcloud.com cms]\> python -c 'import locale;print locale.getdefaultlocale()'
('en_US', 'UTF-8')
(virtenv)[rv-tdf.rhcloud.com cms]\> python -c 'import sys;print sys.getfilesystemencoding()'
UTF-8

Do you think it is possible to change the default /etc/apache2/envars a bit?
It well may be that it would be sufficient to either uncomment the ". /etc/default/locale" line or to add the following pair lines to the configuration file:

export LANG='en_US.UTF-8'
export LC_ALL='en_US.UTF-8'

Version-Release number of selected component (if applicable):

How reproducible:

You may try to upload a non-ascii filename with the Django storage system with the current apache settings. For example, visit the following link:

http://rv-tdf.rhcloud.com/gallery/

Steps to Reproduce:
1. Visit the http://rv-tdf.rhcloud.com/gallery/ page
2. Enjoy django's traceback
3.

Actual results:

Ugly traceback..

Expected results:

A nice picture gallery =)

Additional info:

Thank you! And sorry for my English =)

Best regards,
Dmitry

Comment 1 Maciej Szulik 2014-08-18 08:47:40 UTC
Dmitry, can you please provide us more information regarding your stack? Meaning which python cartridge, which db if any, what is your application, is there a source code available, if not it's ok too. What exactly you put in which files? I want to reproduce your problem on devenv. 
From the traceback I'm assuming following:
python-2.7 cartridge is used, with mezzanine, but is it bare mezzanine or with some addons, if yes - which? 

Thanks,

Comment 2 dmt.kir@gmail.com 2014-08-18 09:16:55 UTC
According to the app console I'm using the following cartridges:
Python 2.7
Web Load Balancer
MySQL 5.5

Actually, I do not use MySQL because every python manage.py createdb|syncdb failed silently (same issue for django-cms as well). I mean, according to the output of the mentioned commands db should be filled with data, but it remained empty every time. I switched to "dev.db" sqlite3 database for now and will try to find a workaround later (it must be a matter of some settings tweak, I suppose).

Also it is pure mezzanine:
$ pip install mezzanine
$ mezzanine-project cms
$ git commit/push
$ ssh to the python-2.7 gear (rhc app show rv --gears)
$ source $OPENSHIFT_HOMEDIR/python/virtenv/bin/activate
$ cd $OPENSHIFT_REPO_DIR/wsgi/
$ cd cms 
$ python manage.py createdb # it works well for the default sqlite3 db engine setting with database name "dev.db"

My wsgi/application:
#!/usr/bin/env python
import os
import sys
import django.core.handlers.wsgi
virtenv = os.environ['APPDIR'] + '/virtenv/'
os.environ['PYTHON_EGG_CACHE'] = os.path.join(virtenv, 'lib/python2.7/site-packages')
virtualenv = os.path.join(virtenv, 'bin/activate_this.py')
try:
    execfile(virtualenv, dict(__file__=virtualenv))
except:
    pass
os.environ['DJANGO_SETTINGS_MODULE'] = 'cms.settings'
sys.path.append(os.path.join(os.environ['OPENSHIFT_REPO_DIR'], 'wsgi', os.environ['OPENSHIFT_APP_NAME']))
application = django.core.handlers.wsgi.WSGIHandler()

+ a few additions to the settings.py:
if 'OPENSHIFT_REPO_DIR' in os.environ:
    STATIC_ROOT = os.path.join(os.environ.get('OPENSHIFT_REPO_DIR'), 'wsgi', 'static')
else:
    STATIC_ROOT = os.path.join(PROJECT_ROOT, STATIC_URL.strip("/"))

if 'OPENSHIFT_DATA_DIR' in os.environ:
    MEDIA_ROOT = os.path.join(os.environ.get('OPENSHIFT_REPO_DIR'), 'wsgi', 'static', 'media')
else:
    MEDIA_ROOT = os.path.join(STATIC_ROOT, 'media')


Thank you!

Comment 3 Maciej Szulik 2014-08-20 13:02:19 UTC
Dmitry, while working with this bug I've created quickstart for mezzanine [1] which smoothly runs against postgresql and mysql, though the latter I haven't tested by myself. Have a look at it, maybe you'll find an answer to your problem with DB. Specifically look into settings.py (section DATABASES) and .openshift/action_hooks/deploy which runs manage.py syncdb.

[1] https://github.com/openshift-quickstart/mezzanine-quickstart/

Comment 4 dmt.kir@gmail.com 2014-08-20 19:30:00 UTC
(In reply to Maciej Szulik from comment #3)
> Dmitry, while working with this bug I've created quickstart for mezzanine
> [1] which smoothly runs against postgresql and mysql, though the latter I
> haven't tested by myself. Have a look at it, maybe you'll find an answer to
> your problem with DB. Specifically look into settings.py (section DATABASES)
> and .openshift/action_hooks/deploy which runs manage.py syncdb.
> 
> [1] https://github.com/openshift-quickstart/mezzanine-quickstart/

Thank you for your help! I really appreciate it :)

Comment 5 Maciej Szulik 2014-08-21 10:25:43 UTC
Dmitry, I'm still investigating the possibility to extend cartridge with these vars. What you can do so far is to define those 2 variables using rhc and then do app stop & start for for this vars to be picked up by apache. So:

rhc set-env LC_ALL='en_US.UTF-8'
rhc set-env LANG='en_US.UTF-8'
rhc app stop
rhc app start

This way you'll be able to upload non-ascii filenames. Unfortunately there's still problem when using 'Zip import' on the page edit, it's also throwing encoding error - still under investigation.

Comment 6 Maciej Szulik 2014-08-21 12:46:36 UTC
Dmitry, the problem with 'Zip import' is in mezzanine itself see [1]. 
We've decided not to add these variables to cart so I'm passing this to QA for confirmation. 

@QA see especially #c5 for resolution.

[1] https://github.com/stephenmcd/mezzanine/issues/1102

Comment 7 Wenjing Zheng 2014-08-22 08:26:36 UTC
Verified with the workaround of comment #5, zip file with non-ascii strings in file name can be uploaded via Zip import.