Bug 1128837 - Dashboard failed to load consoles for instances
Summary: Dashboard failed to load consoles for instances
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-django-horizon
Version: 5.0 (RHEL 7)
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
: 5.0 (RHEL 7)
Assignee: Julie Pichon
QA Contact: Ido Ovadia
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-08-11 15:34 UTC by Jean-Tsung Hsiao
Modified: 2015-02-15 22:02 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-08-18 07:59:37 UTC


Attachments (Terms of Use)
gzip file for horizon.log (189.03 KB, application/x-gzip)
2014-08-15 15:28 UTC, Jean-Tsung Hsiao
no flags Details

Description Jean-Tsung Hsiao 2014-08-11 15:34:38 UTC
Description of problem: Dashboard failed to load consoles for instances

Used to work; but failed lately.

This could be due to a kernel change --- from kernel-3.10.0-138 to kernel-3.10.0-123.6.3.

Version-Release number of selected component (if applicable):


How reproducible: reproducble


Steps to Reproduce:
1. Access dashbord
2. Choose an instance and load console
3.

Actual results:
Instance's console couldn't be loaded.

Expected results:
Should be able to load Instance's console.

Additional info:

Comment 2 Julie Pichon 2014-08-15 13:41:21 UTC
Thank you for the bug report. Some additional information, as well as the versions for the Horizon/novnc packages you're using would be helpful.

Can you access the console in full screen or is it only an issue when the console is embedded in the web page?

What's the error message? Does the console load but cannot connect (grey bar on top shows an error message) or is it a browser error?

Are the nova-novnc, nova-consoleauth services up and running? It probably would be useful to see their logs too. Is the firewall correctly configured to allow access to the VNC console?

Was anything else beside the kernel upgraded?

Thank you.

Comment 3 Jean-Tsung Hsiao 2014-08-15 15:27:18 UTC
(In reply to Julie Pichon from comment #2)
> Thank you for the bug report. Some additional information, as well as the
> versions for the Horizon/novnc packages you're using would be helpful.
[root@qe-dell-ovs2 saveaudit]# rpm -qa | grep -i horizon
python-django-horizon-2014.1.1-2.el7ost.noarch

[root@qe-dell-ovs2 saveaudit]# rpm -qa | grep -i novnc
novnc-0.4-7.el7ost.noarch
openstack-nova-novncproxy-2014.1.1-4.el7ost.noarch
[root@qe-dell-ovs2 saveaudit]# rpm -qa | grep -i dashboard
openstack-dashboard-theme-2014.1.1-2.el7ost.noarch
openstack-dashboard-2014.1.1-2.el7ost.noarch

> 
> Can you access the console in full screen or is it only an issue when the
> console is embedded in the web page?

Chose an instance, click on console, but console loading failed on timeout with following error message:
console is currently unavailable. Please try again later. Reload

> 
> What's the error message? Does the console load but cannot connect (grey bar
> on top shows an error message) or is it a browser error?

See above

> 
> Are the nova-novnc, nova-consoleauth services up and running? It probably
> would be useful to see their logs too. Is the firewall correctly configured
> to allow access to the VNC console?

== Horizon service ==
openstack-dashboard:                    active

Didn't see nova-novnc and nova-consoleauth listed as services:

[root@qe-dell-ovs2 yum.repos.d]# systemctl status nova-novnc
nova-novnc.service
   Loaded: not-found (Reason: No such file or directory)
   Active: inactive (dead)

[root@qe-dell-ovs2 yum.repos.d]# which nova-novnc
/usr/bin/which: no nova-novnc in (/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin)
[root@qe-dell-ovs2 yum.repos.d]# systemctl status nova-consoleauth
nova-consoleauth.service
   Loaded: not-found (Reason: No such file or directory)
   Active: inactive (dead)

[root@qe-dell-ovs2 yum.repos.d]# which nova-consoleauth
/usr/bin/nova-consoleauth

See horizon.log.gz attached.

iptables has been stopped.

> 
> Was anything else beside the kernel upgraded?
> 
> Thank you.

Comment 4 Jean-Tsung Hsiao 2014-08-15 15:28:34 UTC
Created attachment 927144 [details]
gzip file for horizon.log

Comment 5 Julie Pichon 2014-08-15 16:31:17 UTC
Thank you for the information!

(In reply to Jean-Tsung Hsiao from comment #3)
> (In reply to Julie Pichon from comment #2)
> [root@qe-dell-ovs2 saveaudit]# rpm -qa | grep -i novnc
> novnc-0.4-7.el7ost.noarch
> openstack-nova-novncproxy-2014.1.1-4.el7ost.noarch

My novnc version seems a bit more recent, not sure if that would be causing this problem though (novnc-0.4-8.1.el6.noarch).

> > Can you access the console in full screen or is it only an issue when the
> > console is embedded in the web page?
> 
> Chose an instance, click on console, but console loading failed on timeout
> with following error message:
> console is currently unavailable. Please try again later. Reload

Could you provide a screenshot?
What's the status of the instance you're trying to access? (Can you ssh into it, is it active and running?)

> > Are the nova-novnc, nova-consoleauth services up and running? It probably
> > would be useful to see their logs too. Is the firewall correctly configured
> > to allow access to the VNC console?
> 
> == Horizon service ==
> openstack-dashboard:                    active
> 
> Didn't see nova-novnc and nova-consoleauth listed as services:

Oh, they don't seem to be in openstack-status indeed. You should have something like this:

$ ps -ef | grep consoleauth
nova      4127     1  2 16:46 ?        00:00:01 /usr/bin/python /usr/bin/nova-consoleauth --logfile /var/log/nova/consoleauth.log
$ ps -ef | grep novnc
nova      4155     1  0 16:46 ?        00:00:00 /usr/bin/python /usr/bin/nova-novncproxy --web /usr/share/novnc/

$ service openstack-nova-consoleauth status
openstack-nova-consoleauth (pid  7475) is running...
$ service openstack-nova-novncproxy status
openstack-nova-novncproxy (pid  7433) is running...

I would be interested in seeing these services' logs as well. (I see consoleauth.log under /var/log/nova, not sure about the novnc proxy though.) If they were running, could you try restarting them and then see if it helps with loading the console?

I upgraded the kernel and rebooted my 5.0 environment and can still access the console so I'm not quite able to reproduce the issue at this time (I have other problems with novnc, but it seems your problem occurs before even loading the novncproxy page).

Comment 6 Jean-Tsung Hsiao 2014-08-15 17:03:28 UTC
(In reply to Julie Pichon from comment #5)
> Thank you for the information!
> 
> (In reply to Jean-Tsung Hsiao from comment #3)
> > (In reply to Julie Pichon from comment #2)
> > [root@qe-dell-ovs2 saveaudit]# rpm -qa | grep -i novnc
> > novnc-0.4-7.el7ost.noarch
> > openstack-nova-novncproxy-2014.1.1-4.el7ost.noarch
> 
> My novnc version seems a bit more recent, not sure if that would be causing
> this problem though (novnc-0.4-8.1.el6.noarch).
> 
> > > Can you access the console in full screen or is it only an issue when the
> > > console is embedded in the web page?
> > 
> > Chose an instance, click on console, but console loading failed on timeout
> > with following error message:
> > console is currently unavailable. Please try again later. Reload
> 
> Could you provide a screenshot?
> What's the status of the instance you're trying to access? (Can you ssh into
> it, is it active and running?)
> 
> > > Are the nova-novnc, nova-consoleauth services up and running? It probably
> > > would be useful to see their logs too. Is the firewall correctly configured
> > > to allow access to the VNC console?
> > 
> > == Horizon service ==
> > openstack-dashboard:                    active
> > 
> > Didn't see nova-novnc and nova-consoleauth listed as services:
> 
> Oh, they don't seem to be in openstack-status indeed. You should have
> something like this:
> 
> $ ps -ef | grep consoleauth
> nova      4127     1  2 16:46 ?        00:00:01 /usr/bin/python
> /usr/bin/nova-consoleauth --logfile /var/log/nova/consoleauth.log
> $ ps -ef | grep novnc
> nova      4155     1  0 16:46 ?        00:00:00 /usr/bin/python
> /usr/bin/nova-novncproxy --web /usr/share/novnc/
> 
> $ service openstack-nova-consoleauth status
> openstack-nova-consoleauth (pid  7475) is running...
> $ service openstack-nova-novncproxy status
> openstack-nova-novncproxy (pid  7433) is running...
> 
> I would be interested in seeing these services' logs as well. (I see
> consoleauth.log under /var/log/nova, not sure about the novnc proxy though.)
> If they were running, could you try restarting them and then see if it helps
> with loading the console?
> 
> I upgraded the kernel and rebootopenstack-nova-consoleauthed my 5.0 environment and can still access
> the console so I'm not quite able to reproduce the issue at this time (I
> have other problems with novnc, but it seems your problem occurs before even
> loading the novncproxy page).

Something fundamentally went wrong as I moved from 7.1 to 7.0.z. The nova-consoleauth.log stopped at July 29. Service not got enabled and started since then.

-rw-r--r--. 1 nova nova  257236 Jul 29 15:32 nova-consoleauth.log

Just manually started openstack-nova-consoleauth and openstack-nova-novncproxy, and now I can access the instance console as before.

Thanks for looking into this.

NOTABUG!

Comment 7 Julie Pichon 2014-08-18 07:59:37 UTC
(In reply to Jean-Tsung Hsiao from comment #6)
> Just manually started openstack-nova-consoleauth and
> openstack-nova-novncproxy, and now I can access the instance console as
> before.
> 
> Thanks for looking into this.
> 
> NOTABUG!

Excellent. Thanks for the update!


Note You need to log in before you can comment on or make changes to this bug.