Hide Forgot
On Qubes R4.0 with Xen 4.8, I'm facing a problem which is kind of random. Sometimes, a default standard PVH domain, got in it's log ```libxl-driver.log```: ``` 2019-07-09 20:32:04.473+0000: libxl: libxl.c:1853:libxl_console_get_tty: unable to read console tty path `/local/domain/38/console/tty': Ressource temporairement non disponible ``` saying that the tty console is inaccessible whereas in xenstore: ``` 38 = "" device = "" ... console = "" 0 = "" frontend = "/local/domain/38/console" backend = "/local/domain/0/backend/console/38/0" frontend-id = "38" online = "1" state = "1" protocol = "vt100" type = "pvh" dm-version = "qemu_xen" ``` it exists and is accessible throught ```xl console```. A look at the current libvirt XML part related to the console of the VM: ``` <console type='pty'> <target type='xen' port='0'/> </console> ``` the tty ```/dev/pts/X``` is not defined. Shutting down all the running VM and starting again this VM allows to obtain finally a tty console through libvirt. It seems to be on libvirt side and it's possibly a race condition. I never experienced such behavior. I'm experimenting this problem on a HP DL360p Gen8 server. We issued this on our tracker too: https://github.com/QubesOS/qubes-issues/issues/5156. If someone has any idea or patch to be tested, I can do that easily. Thank you for your help.
Adding Jim to CC, since he's the most familiar with Libvirt's libxl support
(In reply to Frédéric Pierret from comment #0) > On Qubes R4.0 with Xen 4.8, I'm facing a problem which is kind of random. What version of libvirt? > 2019-07-09 20:32:04.473+0000: libxl: libxl.c:1853:libxl_console_get_tty: > unable to read console tty path `/local/domain/38/console/tty': Ressource > temporairement non disponible > ``` > saying that the tty console is inaccessible whereas in xenstore: > ``` > 38 = "" > device = "" > ... > console = "" > 0 = "" > frontend = "/local/domain/38/console" > backend = "/local/domain/0/backend/console/38/0" > frontend-id = "38" > online = "1" > state = "1" > protocol = "vt100" > type = "pvh" > dm-version = "qemu_xen" That snippet of xenstore output does not include the path /local/domain/38/console/tty. It looks to be from the /libxl/<domid> path. Do you really have a pts in /local/domain/<domid>/console/tty when the problem occurs? E.g. what is the output of 'xenstore-read /local/domain/<domid>/console/tty'?
(In reply to Jim Fehlig from comment #2) > (In reply to Frédéric Pierret from comment #0) > > On Qubes R4.0 with Xen 4.8, I'm facing a problem which is kind of random. > > What version of libvirt? It's libvirt 3.3.0. > > 2019-07-09 20:32:04.473+0000: libxl: libxl.c:1853:libxl_console_get_tty: > > unable to read console tty path `/local/domain/38/console/tty': Ressource > > temporairement non disponible > > ``` > > saying that the tty console is inaccessible whereas in xenstore: > > ``` > > 38 = "" > > device = "" > > ... > > console = "" > > 0 = "" > > frontend = "/local/domain/38/console" > > backend = "/local/domain/0/backend/console/38/0" > > frontend-id = "38" > > online = "1" > > state = "1" > > protocol = "vt100" > > type = "pvh" > > dm-version = "qemu_xen" > > That snippet of xenstore output does not include the path > /local/domain/38/console/tty. It looks to be from the /libxl/<domid> path. > Do you really have a pts in /local/domain/<domid>/console/tty when the > problem occurs? E.g. what is the output of 'xenstore-read > /local/domain/<domid>/console/tty'? Yes I got a console because I can do a 'xl console' and it's working. Notably, a 'xenstore-read /local/domain/31/console/tty' returns '/dev/pts/6' (for another domain since the current posted but with the same info).
Your xen and libvirt are fairly old. Are you able to try newer versions? FYI, SLES12 SP3 has a fair bit of xen 4.9.x + libvirt 3.3.0 users and I'm not aware of any console issues. As you say there is a race condition between libvirt's console callback being invoked and its subsequent reading of /local/domain/<domid>/console/tty via the libxl_console_get_tty() API. It seems more like a problem in xen IMO. libxl should not be invoking the callback if the tty is not ready and /local/domain/<domid>/console/tty is not set. Also, from your xenstore snippet I see you are using 'pvh' type machine. libvirt did not gain support for pvh machines until version 4.10.0. Is your libvirt patched to support pvh?
(In reply to Jim Fehlig from comment #4) > Your xen and libvirt are fairly old. Are you able to try newer versions? > FYI, SLES12 SP3 has a fair bit of xen 4.9.x + libvirt 3.3.0 users and I'm > not aware of any console issues. I can try newer version with Xen 4.12 and libvirt 5.0.0. That would require a little bit of time for that (time to setup our latest Qubes development version). > As you say there is a race condition between libvirt's console callback > being invoked and its subsequent reading of > /local/domain/<domid>/console/tty via the libxl_console_get_tty() API. It > seems more like a problem in xen IMO. libxl should not be invoking the > callback if the tty is not ready and /local/domain/<domid>/console/tty is > not set. > > Also, from your xenstore snippet I see you are using 'pvh' type machine. > libvirt did not gain support for pvh machines until version 4.10.0. Is your > libvirt patched to support pvh? Yes indeed, you can see the patches here: https://github.com/QubesOS/qubes-core-libvirt/tree/release4.0