Bug 1731678 - libvirt: ttyconsole not created
Summary: libvirt: ttyconsole not created
Keywords:
Status: NEW
Alias: None
Product: Virtualization Tools
Classification: Community
Component: libvirt
Version: unspecified
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: ---
Assignee: Libvirt Maintainers
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-07-21 08:28 UTC by Frédéric Pierret
Modified: 2019-07-23 15:59 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:


Attachments (Terms of Use)

Description Frédéric Pierret 2019-07-21 08:28:39 UTC
On Qubes R4.0 with Xen 4.8, I'm facing a problem which is kind of random. Sometimes, a default standard PVH domain, got in it's log ```libxl-driver.log```:
```
2019-07-09 20:32:04.473+0000: libxl: libxl.c:1853:libxl_console_get_tty: unable to read console tty path `/local/domain/38/console/tty': Ressource temporairement non disponible
```
saying that the tty console is inaccessible whereas in xenstore:
```
38 = ""
  device = ""
...
   console = ""
    0 = ""
     frontend = "/local/domain/38/console"
     backend = "/local/domain/0/backend/console/38/0"
     frontend-id = "38"
     online = "1"
     state = "1"
     protocol = "vt100"
  type = "pvh"
  dm-version = "qemu_xen"
```
it exists and is accessible throught ```xl console```. A look at the current libvirt XML part related to the console of the VM:
```
<console type='pty'>
  <target type='xen' port='0'/>
</console>
```
the tty ```/dev/pts/X``` is not defined. Shutting down all the running VM and starting again this VM allows to obtain finally a tty console through libvirt. It seems to be on libvirt side and it's possibly a race condition. I never experienced such behavior. I'm experimenting this problem on a HP DL360p Gen8 server.

We issued this on our tracker too: https://github.com/QubesOS/qubes-issues/issues/5156.

If someone has any idea or patch to be tested, I can do that easily. Thank you for your help.

Comment 1 Daniel Berrangé 2019-07-22 09:01:36 UTC
Adding Jim to CC, since he's the most familiar with Libvirt's libxl support

Comment 2 Jim Fehlig 2019-07-22 14:29:20 UTC
(In reply to Frédéric Pierret from comment #0)
> On Qubes R4.0 with Xen 4.8, I'm facing a problem which is kind of random.

What version of libvirt?

> 2019-07-09 20:32:04.473+0000: libxl: libxl.c:1853:libxl_console_get_tty:
> unable to read console tty path `/local/domain/38/console/tty': Ressource
> temporairement non disponible
> ```
> saying that the tty console is inaccessible whereas in xenstore:
> ```
> 38 = ""
>   device = ""
> ...
>    console = ""
>     0 = ""
>      frontend = "/local/domain/38/console"
>      backend = "/local/domain/0/backend/console/38/0"
>      frontend-id = "38"
>      online = "1"
>      state = "1"
>      protocol = "vt100"
>   type = "pvh"
>   dm-version = "qemu_xen"

That snippet of xenstore output does not include the path /local/domain/38/console/tty. It looks to be from the /libxl/<domid> path. Do you really have a pts in /local/domain/<domid>/console/tty when the problem occurs? E.g. what is the output of 'xenstore-read /local/domain/<domid>/console/tty'?

Comment 3 Frédéric Pierret 2019-07-23 14:09:12 UTC
(In reply to Jim Fehlig from comment #2)
> (In reply to Frédéric Pierret from comment #0)
> > On Qubes R4.0 with Xen 4.8, I'm facing a problem which is kind of random.
> 
> What version of libvirt?

It's libvirt 3.3.0.

> > 2019-07-09 20:32:04.473+0000: libxl: libxl.c:1853:libxl_console_get_tty:
> > unable to read console tty path `/local/domain/38/console/tty': Ressource
> > temporairement non disponible
> > ```
> > saying that the tty console is inaccessible whereas in xenstore:
> > ```
> > 38 = ""
> >   device = ""
> > ...
> >    console = ""
> >     0 = ""
> >      frontend = "/local/domain/38/console"
> >      backend = "/local/domain/0/backend/console/38/0"
> >      frontend-id = "38"
> >      online = "1"
> >      state = "1"
> >      protocol = "vt100"
> >   type = "pvh"
> >   dm-version = "qemu_xen"
> 
> That snippet of xenstore output does not include the path
> /local/domain/38/console/tty. It looks to be from the /libxl/<domid> path.
> Do you really have a pts in /local/domain/<domid>/console/tty when the
> problem occurs? E.g. what is the output of 'xenstore-read
> /local/domain/<domid>/console/tty'?

Yes I got a console because I can do a 'xl console' and it's working. Notably, a 'xenstore-read /local/domain/31/console/tty' returns '/dev/pts/6' (for another domain since the current posted but with the same info).

Comment 4 Jim Fehlig 2019-07-23 15:54:07 UTC
Your xen and libvirt are fairly old. Are you able to try newer versions? FYI, SLES12 SP3 has a fair bit of xen 4.9.x + libvirt 3.3.0 users and I'm not aware of any console issues.

As you say there is a race condition between libvirt's console callback being invoked and its subsequent reading of /local/domain/<domid>/console/tty via the libxl_console_get_tty() API. It seems more like a problem in xen IMO. libxl should not be invoking the callback if the tty is not ready and /local/domain/<domid>/console/tty is not set.

Also, from your xenstore snippet I see you are using 'pvh' type machine. libvirt did not gain support for pvh machines until version 4.10.0. Is your libvirt patched to support pvh?

Comment 5 Frédéric Pierret 2019-07-23 15:59:57 UTC
(In reply to Jim Fehlig from comment #4)
> Your xen and libvirt are fairly old. Are you able to try newer versions?
> FYI, SLES12 SP3 has a fair bit of xen 4.9.x + libvirt 3.3.0 users and I'm
> not aware of any console issues.

I can try newer version with Xen 4.12 and libvirt 5.0.0. That would require a little bit of time for that (time to setup our latest Qubes development version).

> As you say there is a race condition between libvirt's console callback
> being invoked and its subsequent reading of
> /local/domain/<domid>/console/tty via the libxl_console_get_tty() API. It
> seems more like a problem in xen IMO. libxl should not be invoking the
> callback if the tty is not ready and /local/domain/<domid>/console/tty is
> not set.
> 
> Also, from your xenstore snippet I see you are using 'pvh' type machine.
> libvirt did not gain support for pvh machines until version 4.10.0. Is your
> libvirt patched to support pvh?

Yes indeed, you can see the patches here: https://github.com/QubesOS/qubes-core-libvirt/tree/release4.0


Note You need to log in before you can comment on or make changes to this bug.