Bug 1352620

Summary: qemu-kvm fail to start in vnc reverse mode
Product: Red Hat Enterprise Linux 7 Reporter: Guo, Zhiyi <zhguo>
Component: qemu-kvm-rhevAssignee: Gerd Hoffmann <kraxel>
Status: CLOSED ERRATA QA Contact: Guo, Zhiyi <zhguo>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 7.3CC: areis, chayang, jinzhao, juzhang, knoel, kraxel, michen, mrezanin, red, virt-maint, xfu, zhguo
Target Milestone: rcKeywords: Regression
Target Release: ---   
Hardware: x86_64   
OS: Unspecified   
Whiteboard:
Fixed In Version: qemu-kvm-rhev-2.9.0-1.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-08-01 23:32:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Guo, Zhiyi 2016-07-04 12:46:35 UTC
Description of problem:
qemu-kvm fail to start in vnc reverse mode

Version-Release number of selected component (if applicable):
qemu-kvm:
qemu-kvm-rhev-2.6.0-11.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.launch a listening vnc on host A by:
# vncviewer -listen 7000
2.launch qemu-kvm process and try to connect to the listening vnc on host B by:
...-vnc ${ip_of_host_A's}:7000,"reverse"

Actual results:
From host A(ip:10.72.4.72):
vncviewer --listen 7000

TigerVNC Viewer 64-bit v1.6.0
Built on: 2016-01-04 15:10
Copyright (C) 1999-2015 TigerVNC Team and many others (see README.txt)
See http://www.tigervnc.org for information on TigerVNC.

Mon Jul  4 17:57:15 2016
 main:        Listening on port 7000

From host B:
# /usr/libexec/qemu-kvm -vnc 10.72.4.72:7000,reverse
warning: host doesn't support requested feature: CPUID.80000001H:ECX.abm [bit 5]
warning: host doesn't support requested feature: CPUID.80000001H:ECX.sse4a [bit 6]
qemu-kvm: -vnc 10.72.4.72:7000,reverse: Failed to start VNC server: Failed to connect socket: Connection refused

Expected results:
qemu-kvm start in vnc reverse mode 

Additional info:
No such issue happen on qemu-kvm-rhev version:
qemu-kvm-rhev-2.3.0-31.el7_2.15.x86_64

Comment 4 Gerd Hoffmann 2016-08-11 12:50:33 UTC
possibly bug 1359655 duplicate, please test this scratch build:
https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=11556084

Comment 5 Guo, Zhiyi 2016-08-16 11:37:02 UTC
Test against qemu package:qemu-kvm-rhev-2.6.0-20.el7.bz1359655.2.x86_64, issue still happen.

This may provide some hints for you:
Test against qemu-kvm-rhev-2.3.0-31.el7_2.21.x86_64 which not hit the issue:
1.lauch vncviewer in listen mode on host A(ip 10.66.9.154):
vncviewer -listen 5500 
2.launch qemu-kvm with strace:
strace /usr/libexec/qemu-kvm -vnc 10.66.9.154:5500,reverse
reverse mode works well and I got:
connect(10, {sa_family=AF_INET, sin_port=htons(5500), sin_addr=inet_addr("10.66.9.154")}, 16) = 0
fcntl(10, F_GETFL)                      = 0x2 (flags O_RDWR)
fcntl(10, F_SETFL, O_RDWR|O_NONBLOCK)   = 0
getpeername(10, {sa_family=AF_INET, sin_port=htons(5500), sin_addr=inet_addr("10.66.9.154")}, [16]) = 0

Test above steps 10 times and I always see htons(5500) happen and reverse mode can connect

Test against qemu-kvm-rhev-2.6.0-20.el7.bz1359655.2.x86_64 with same steps, issue still happen and I got trace and seems port is different(11400 vs 5500):
connect(13, {sa_family=AF_INET, sin_port=htons(11400), sin_addr=inet_addr("10.66.9.154")}, 16) = -1 ECONNREFUSED (Connection refused)

When I launch vncviewer in "vncviewer -listen 11400" and use "strace /usr/libexec/qemu-kvm -vnc 10.66.9.154:11400,reverse", port will jump to another:
connect(13, {sa_family=AF_INET, sin_port=htons(17300), sin_addr=inet_addr("10.66.9.154")}, 16) = -1 ECONNREFUSED (Connection refused)

Comment 6 Guo, Zhiyi 2016-08-16 12:08:09 UTC
Try to use port from htons(port) to connect reverse mode,eg:
1.lauch vncviewer in listen mode on host A(ip 10.66.9.154):
vncviewer -listen 11400
2.launch qemu-kvm with strace:
strace /usr/libexec/qemu-kvm -vnc 10.66.9.154:5500,reverse

qemu will coredump and trace output from strace and gdb:
trace:
connect(13, {sa_family=AF_INET, sin_port=htons(11400), sin_addr=inet_addr("10.66.9.154")}, 16) = 0
getpeername(13, {sa_family=AF_INET, sin_port=htons(11400), sin_addr=inet_addr("10.66.9.154")}, [16]) = 0
getsockname(13, {sa_family=AF_INET, sin_port=htons(48188), sin_addr=inet_addr("10.66.9.154")}, [16]) = 0
getsockopt(13, SOL_SOCKET, SO_ACCEPTCONN, [0], [4]) = 0
fcntl(13, F_GETFL)                      = 0x2 (flags O_RDWR)
fcntl(13, F_SETFL, O_RDWR|O_NONBLOCK)   = 0
write(9, "\1\0\0\0\0\0\0\0", 8)         = 8
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0xb8} ---
+++ killed by SIGSEGV (core dumped) +++
Segmentation fault (core dumped)

gdb:
(gdb) bt
#0  qio_channel_socket_get_local_address (ioc=0x0, errp=errp@entry=0x7ffdf2769f90) at io/channel-socket.c:33
#1  0x00007f89ebc5d56f in vnc_init_basic_info_from_server_addr (errp=0x7ffdf2769f90, info=0x7f89f03f4860, ioc=<optimized out>)
    at ui/vnc.c:146
#2  vnc_server_info_get (vd=0x7f89f030c000) at ui/vnc.c:223
#3  0x00007f89ebc62b28 in vnc_qmp_event (vs=0x7f89f0b28000, vs=0x7f89f0b28000, event=QAPI_EVENT_VNC_CONNECTED) at ui/vnc.c:279
#4  vnc_connect (vd=vd@entry=0x7f89f030c000, sioc=sioc@entry=0x7f89f0b75340, skipauth=skipauth@entry=false, websocket=websocket@entry=false)
    at ui/vnc.c:2999
#5  0x00007f89ebc632dc in vnc_display_open (id=id@entry=0x7f89eeb6a528 "default", errp=errp@entry=0x7ffdf276a0d0) at ui/vnc.c:3778
#6  0x00007f89ebc64200 in vnc_init_func (opaque=<optimized out>, opts=<optimized out>, errp=<optimized out>) at ui/vnc.c:3873
#7  0x00007f89ebd1fb6a in qemu_opts_foreach (list=<optimized out>, func=0x7f89ebc641c0 <vnc_init_func>, opaque=opaque@entry=0x0,
    errp=errp@entry=0x0) at util/qemu-option.c:1116
#8  0x00007f89eba4d0d2 in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4615

Comment 7 Gerd Hoffmann 2016-09-05 09:03:50 UTC
reverse mode is unsupported, so low priority and doesn't qualify as blocker.
lets move to 7.4, hopefully picking up whatever upstream does.

Comment 9 Gerd Hoffmann 2017-01-09 15:19:10 UTC
IIRC this was fixed upstream, can't find the commit though.
Can you re-test a qemu-2.8.0 build please?

Comment 10 Guo, Zhiyi 2017-01-11 07:59:50 UTC
Pull latest qemu stream code from qemu.org

switch to 2.8.0 and check tag info:
# git show 834f262
2.8.0

Configure qemu with below features and compile pass:
./configure --disable-strip --disable-qom-cast-debug '--extra-ldflags=-Wl,--build-id -pie -Wl,-z,relro -Wl,-z,now' '--extra-cflags=-O2 -g -pipe -Wall  -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIE -DPIE' --enable-trace-backend=dtrace --enable-werror --disable-xen --disable-virtfs --enable-kvm --enable-libusb --enable-spice --enable-seccomp --disable-fdt --enable-docs --disable-sdl --disable-debug-tcg --disable-sparse --disable-brlapi --disable-bluez --disable-vde --disable-curses --enable-curl --enable-libssh2 --enable-vnc --enable-vnc-sasl --enable-linux-aio  --enable-lzo --enable-snappy --enable-usb-redir --enable-vnc-png --disable-vnc-jpeg  --enable-uuid --disable-vhost-scsi --disable-guest-agent   --enable-rbd --enable-glusterfs --enable-tcmalloc --block-drv-rw-whitelist=qcow2,raw,file,host_device,blkdebug,nbd,iscsi,gluster,rbd --block-drv-ro-whitelist=vmdk,vhdx,vpc,ssh,https --target-list=x86_64-softmmu

Start vncviewer in listen mode by:
vncviewer --listen 7000
and try to connect to vncviewer using reverse mode, reverse mode fail to start, ip used here has vncviewer running:
./qemu-system-x86_64 -vnc 10.66.8.190:7000,reverse
qemu-system-x86_64: -vnc 10.66.8.190:7000,reverse: Failed to start VNC server: Failed to connect socket: Connection refused

So reverse mode still not include in qemu 2.8?

Comment 11 Gerd Hoffmann 2017-01-11 08:49:36 UTC
Hmm, the problem is elsewhere.  Old qemu versions interpret the number in "-vnc $host:$nr,reverse" as port number.  Newer qemu versions interpret the number as display number (i.e. with 5900 offset).

Try "vncviewer --listen 5904" and "qemu -vnc localhost:4,reverse", that should work fine.

Question is what to do about it now.  IMO the new behavior is more consistent ($nr is interpreted as display number in both reverse and normal mode).  It is a incompatible change, but as reverse mode is not supported anyway so we can live with it I think.

So CLOSE/WONTFIX?  Or CLOSE/NOTABUG?  areis?

Comment 12 Ademar Reis 2017-01-11 14:55:16 UTC
(In reply to Gerd Hoffmann from comment #11)
> Hmm, the problem is elsewhere.  Old qemu versions interpret the number in
> "-vnc $host:$nr,reverse" as port number.  Newer qemu versions interpret the
> number as display number (i.e. with 5900 offset).
> 
> Try "vncviewer --listen 5904" and "qemu -vnc localhost:4,reverse", that
> should work fine.
> 
> Question is what to do about it now.  IMO the new behavior is more
> consistent ($nr is interpreted as display number in both reverse and normal
> mode).  It is a incompatible change, but as reverse mode is not supported
> anyway so we can live with it I think.
> 
> So CLOSE/WONTFIX?  Or CLOSE/NOTABUG?  areis?

I agree with closing it, but we need to fix the qemu manpage:

reverse
      Connect to a listening VNC client via a "reverse" connection. The client is specified by the display. For reverse network connections (host:d,"reverse"), the d argument is a TCP port number, not a display number.

Comment 13 Davide Repetto 2017-02-09 13:38:15 UTC
there is still a problem.

In VNC the reverse "ip-addr:port" has always been a port. Never a display. Because in "reverse" you don't connect to a display. Intead you connect to a vnc viewer, which is listenting on a port, which in turn is not "related" to any "display", simply because "displays" have no meaning whatsoever on the listenting viewer side. They may have a meaning on the server side, but not on a listening viewer.

Because qemu maps display/port parameter wrong when calling "in reverse" to a listening viewer, it maps its ports with an offset starting from 5900, which makes no sense whatsoever, especially because the default port on all vnc viewers capable of listening is 5500.
Thus this makes it impossible to connect to any VNC listening listening server configured on its default port.

So this qemu behaviour is totally confusing, counter intuitive and non convenient.

Comment 14 Gerd Hoffmann 2017-03-14 08:29:49 UTC
https://patchwork.ozlabs.org/patch/738585/

Comment 15 Gerd Hoffmann 2017-03-31 11:45:34 UTC
merged upstream (e5766eb40453b1d22815fc5482802688ff184006).
rebase will pick up.

Comment 17 Chao Yang 2017-05-10 08:24:25 UTC
Reproduced with qemu-kvm-rhev-2.6.0-27.el7

Host A:
vncviewer -listen 

Host B:
/usr/libexec/qemu-kvm -vnc 10.66.8.200:5500,reverse

Result:
With qemu-kvm-rhev-2.6.0-27.el7:
qemu-kvm: -vnc 10.66.8.200:5500,reverse: Failed to start VNC server: Failed to connect socket: Connection refused


With qemu-kvm-rhev-2.9.0-3.el7, connect set up correctly.

Setting VERIFIED accordingly.

Comment 19 errata-xmlrpc 2017-08-01 23:32:13 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392

Comment 20 errata-xmlrpc 2017-08-02 01:09:52 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392

Comment 21 errata-xmlrpc 2017-08-02 02:01:51 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392

Comment 22 errata-xmlrpc 2017-08-02 02:42:37 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392

Comment 23 errata-xmlrpc 2017-08-02 03:07:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392

Comment 24 errata-xmlrpc 2017-08-02 03:27:29 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392