Bug 950903

Summary:	console logs not available for recipes running on dynamic VMs
Product:	[Retired] Beaker	Reporter:	Dan Callaghan <dcallagh>
Component:	scheduler	Assignee:	matt jia <mjia>
Status:	CLOSED CURRENTRELEASE	QA Contact:	tools-bugs <tools-bugs>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	0.11	CC:	aigao, asaha, dcallagh, jingwang, mjia, qwan, rmancy, tools-bugs, xma
Target Milestone:	0.18
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:	Cloud
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2014-09-10 07:02:26 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Dan Callaghan 2013-04-11 07:38:13 UTC

Description of problem:
Serial console logs are not available for recipes running on oVirt/RHEV dynamic VMs.

Version-Release number of selected component (if applicable):
0.12.0

How reproducible:
always

Steps to Reproduce:
1. Schedule a recipe which is virtualisable
2. Watch it run
3. No console.log
  
Additional info:
The console log is necessary to diagnose many kinds of problems, including errors reported by Anaconda, problems with the boot images, a misconfigured VM, failures in the harness, network problems communicating with Beaker...

Comment 1 Dan Callaghan 2013-04-16 05:53:30 UTC

Part of this bug fix should be to fix beaker-watchdog not to look for /var/consoles/None while the VM is installing.

Comment 3 Dan Callaghan 2014-05-16 05:09:35 UTC

This will be an issue when we port to OpenStack as well.

Comment 4 Dan Callaghan 2014-06-23 00:55:57 UTC

OpenStack has an API for fetching the serial console log for an instance (it returns the entire thing). So the first draft of this feature:

* Make instances use the serial console (probably just need to add console=ttyS0 to kernel options for all OpenStack instances)

* Make beaker-watchdog aware of the instance ID for OpenStack instances, and some usable credentials to authenticate to OpenStack

* Instead of reading the console from /var/consoles beaker-watchdog should use the OpenStack API to fetch the entire console log, and run it through the usual processing (wipe control characters, check for panics and install failures, store a copy locally)

Bonus points for:

* Patching OpenStack to support fetching a range of bytes from the log, instead of the whole thing, so that we can more efficiently poll it

* Make it possible to log in to the console even when console=ttyS0 is passed (currently that will make the system use the serial console for /dev/console but OpenStack does not expose a writable serial console which leaves no way to log in)

Comment 5 matt jia 2014-07-07 02:03:37 UTC

I have opened a bug for OpenStack to support fetching a selecting range of console logs.

https://bugs.launchpad.net/nova/+bug/1337695

Cheers,Matt Jia

Comment 6 matt jia 2014-07-09 23:33:51 UTC

On gerrit

http://gerrit.beaker-project.org/#/c/3181/

Cheers, Matt Jia

Comment 7 matt jia 2014-07-13 23:51:36 UTC

OpenStack has a size limit for getting console logs.I have field a bug to increase this which should be big enough for us to get the whole thing.

https://bugs.launchpad.net/nova/+bug/1341363

In the future, we can switch to the interactive web-based serial consoles which they are currently working on.

https://review.openstack.org/#/c/86418/7/specs/juno/serial-ports.rst

Cheers,Matt Jia

Comment 8 matt jia 2014-07-22 06:34:51 UTC

I found a new issue when I upgraded the above fix to beaker-devel.Move back to ASSGINED To get this fix done.

Comment 9 matt jia 2014-07-22 06:48:51 UTC

On Gerrit:

http://gerrit.beaker-project.org/#/c/3215/

Cheers, Matt Jia

Comment 12 Dan Callaghan 2014-09-10 07:02:26 UTC

This was actually in Beaker 0.18.0.