Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1641826

Summary: Environment with pre-provisioned nodes after overcloud / container update returns [Errno 13] Permission denied: '/var/log/horizon/horizon.log' until horizon container is restarted
Product: Red Hat OpenStack Reporter: Andreas Karis <akaris>
Component: openstack-tripleo-heat-templatesAssignee: Bogdan Dobrelya <bdobreli>
Status: CLOSED ERRATA QA Contact: Arik Chernetsky <achernet>
Severity: high Docs Contact:
Priority: high    
Version: 13.0 (Queens)CC: akaris, aschultz, bdobreli, beth.white, david.pasqua, emacchi, jrist, jschluet, knoha, mburns, rdopiera, slinaber, ssigwald
Target Milestone: z6Keywords: TestOnly, Triaged, ZStream
Target Release: 13.0 (Queens)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-8.3.1-5.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-07-10 13:03:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Andreas Karis 2018-10-22 21:57:14 UTC
Description of problem:

Environment with pre-provisioned nodes after overcloud / container update returns [Errno 13] Permission denied: '/var/log/horizon/horizon.log' until horizon container is restarted 

This has been deployed via director using pre-deployed nodes (as per documentation).
~~~
From /var/log/containers/httpd/horizon/horizon_error.log.1: 
[Thu Oct 11 17:54:28.562773 2018] [:error] [pid 59] [remote 10.67.116.65:0] mod_wsgi (pid=59): Target WSGI script '/usr/share/openstack-dashboard/openstack_dashboard/wsgi/django.wsgi' cannot be loaded as Python module.
[Thu Oct 11 17:54:28.562812 2018] [:error] [pid 59] [remote 10.67.116.65:0] mod_wsgi (pid=59): Exception occurred processing WSGI script '/usr/share/openstack-dashboard/openstack_dashboard/wsgi/django.wsgi'.
[Thu Oct 11 17:54:28.562828 2018] [:error] [pid 59] [remote 10.67.116.65:0] Traceback (most recent call last):
[Thu Oct 11 17:54:28.562843 2018] [:error] [pid 59] [remote 10.67.116.65:0]   File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/django.wsgi", line 33, in <module>
[Thu Oct 11 17:54:28.562907 2018] [:error] [pid 59] [remote 10.67.116.65:0]     application = get_wsgi_application()
[Thu Oct 11 17:54:28.562915 2018] [:error] [pid 59] [remote 10.67.116.65:0]   File "/usr/lib/python2.7/site-packages/django/core/wsgi.py", line 13, in get_wsgi_application
[Thu Oct 11 17:54:28.562985 2018] [:error] [pid 59] [remote 10.67.116.65:0]     django.setup(set_prefix=False)
[Thu Oct 11 17:54:28.562995 2018] [:error] [pid 59] [remote 10.67.116.65:0]   File "/usr/lib/python2.7/site-packages/django/__init__.py", line 22, in setup
[Thu Oct 11 17:54:28.563028 2018] [:error] [pid 59] [remote 10.67.116.65:0]     configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
[Thu Oct 11 17:54:28.563035 2018] [:error] [pid 59] [remote 10.67.116.65:0]   File "/usr/lib/python2.7/site-packages/django/utils/log.py", line 75, in configure_logging
[Thu Oct 11 17:54:28.563096 2018] [:error] [pid 59] [remote 10.67.116.65:0]     logging_config_func(logging_settings)
[Thu Oct 11 17:54:28.563104 2018] [:error] [pid 59] [remote 10.67.116.65:0]   File "/usr/lib64/python2.7/logging/config.py", line 803, in dictConfig
[Thu Oct 11 17:54:28.563304 2018] [:error] [pid 59] [remote 10.67.116.65:0]     dictConfigClass(config).configure()
[Thu Oct 11 17:54:28.563313 2018] [:error] [pid 59] [remote 10.67.116.65:0]   File "/usr/lib64/python2.7/logging/config.py", line 585, in configure
[Thu Oct 11 17:54:28.563327 2018] [:error] [pid 59] [remote 10.67.116.65:0]     '%r: %s' % (name, e))
[Thu Oct 11 17:54:28.563341 2018] [:error] [pid 59] [remote 10.67.116.65:0] ValueError: Unable to configure handler 'file': [Errno 13] Permission denied: '/var/log/horizon/horizon.log'
[Thu Oct 11 18:00:26.411913 2018] [:error] [pid 58] [remote 10.67.116.65:124] mod_wsgi (pid=58): Target WSGI script '/usr/share/openstack-dashboard/openstack_dashboard/wsgi/django.wsgi' cannot be loaded as Python module.
[Thu Oct 11 18:00:26.411942 2018] [:error] [pid 58] [remote 10.67.116.65:124] mod_wsgi (pid=58): Exception occurred processing WSGI script '/usr/share/openstack-dashboard/openstack_dashboard/wsgi/django.wsgi'.
~~~

Comment 7 Jason E. Rist 2018-10-25 14:47:41 UTC
Is this part of an upgrade?  Do you expect the container refresh to happen automatically?

Comment 9 Andreas Karis 2018-11-07 23:34:45 UTC
Hi,

Confirmed for a 3rd customer. How to reproduce:

i) run openstack overcloud deploy to push a configuration change via a stack update
ii) reboot controller node

Horizon will not come up. Check error logs:
~~~
[root@overcloud-controller-0 ~]# grep -i permission /var/log/containers/httpd/horizon/horizon_error.log
[Wed Nov 07 23:29:41.155014 2018] [:error] [pid 57] [remote 192.168.102.7:0] ValueError: Unable to configure handler 'file': [Errno 13] Permission denied: '/var/log/horizon/horizon.log'
[Wed Nov 07 23:29:47.932982 2018] [:error] [pid 59] [remote 192.168.102.7:0] ValueError: Unable to configure handler 'file': [Errno 13] Permission denied: '/var/log/horizon/horizon.log'
[Wed Nov 07 23:29:55.489593 2018] [:error] [pid 59] [remote 192.168.102.7:156] ValueError: Unable to configure handler 'file': [Errno 13] Permission denied: '/var/log/horizon/horizon.log'
[root@overcloud-controller-0 ~]# date
Wed  7 Nov 23:32:08 UTC 2018
~~~

iii) Fix the issue:
~~~
[root@overcloud-controller-0 ~]# docker ps | grep horizon
9b1f6065f654        192.168.103.1:8787/rhosp13/openstack-horizon:13.0-54                       "kolla_start"            13 days ago         Up 6 minutes                                   horizon
[root@overcloud-controller-0 ~]# docker restart horizon
horizon
[root@overcloud-controller-0 ~]#
~~~

Comment 10 David Pasqua 2019-03-06 13:43:13 UTC
Hi guys,
Any update on this?
i'm having the same issue on two customers
if you need some logs please let me know

as workaround a put a cron to fix the permissions at container boot time
()[root@overcloud-controller-2 /]# crontab -l
@reboot /root/fixlogperms.sh
()[root@overcloud-controller-2 /]# cat /root/fixlogperms.sh 
chown apache:apache /var/log/horizon/*

Comment 13 Radomir Dopieralski 2019-03-08 12:52:18 UTC
As far as I can see, Horizon has the permissions for the log directory configured properly:

https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/docker/services/horizon.yaml#L88-L91

This looks like a problem with how tripleo itself works, and not with Horizon or its configuration. I'm afraid I'm unable to help with this issue, it needs to go to tripleo's DFG.

Comment 24 errata-xmlrpc 2019-07-10 13:03:16 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1738