Bug 1446003

Summary: vnc cannot find a free port to use
Product: Red Hat Enterprise Linux 7 Reporter: Guo, Zhiyi <zhguo>
Component: qemu-kvm-rhevAssignee: Daniel Berrangé <berrange>
Status: CLOSED ERRATA QA Contact: Guo, Zhiyi <zhguo>
Severity: medium Docs Contact:
Priority: urgent    
Version: 7.4CC: berrange, chayang, jinzhao, juzhang, knoel, kraxel, lmiksik, virt-maint
Target Milestone: rcKeywords: Regression
Target Release: ---   
Hardware: x86_64   
OS: All   
Whiteboard:
Fixed In Version: qemu-kvm-rhev-2.9.0-14.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-08-02 04:35:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Guo, Zhiyi 2017-04-27 00:16:01 UTC
Description of problem:
vnc cannot find a free port to use

Version-Release number of selected component (if applicable):
qemu-kvm-rhev-2.9.0-1.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.boot qemu as /usr/libexec/qemu-kvm -vnc :1,to=2
2.boot another qemu as /usr/libexec/qemu-kvm -vnc :1,to=2
3.

Actual results:
Second qemu fail to start with:
qemu-kvm: -vnc :1,to=2: Failed to start VNC server: Failed to bind socket: Address already in use

Expected results:
Second qemu can start and use free vnc port 2

Additional info:
This is a regression of qemu-kvm 2.9. Compile qemu-kvm from 2.8.0 upstream/downstream source, these can start:
$ ./qemu-system-x86_64 -vnc :0,to=2 -monitor stdio
QEMU 2.8.1.1 monitor - type 'help' for more information
(qemu) info vnc
Server:
     address: 0.0.0.0:5900
        auth: none

./qemu-system-x86_64 -vnc :0,to=2 -monitor stdio
QEMU 2.8.1.1 monitor - type 'help' for more information
(qemu) info vnc
Server:
     address: 0.0.0.0:5901

./qemu-system-x86_64 -vnc :0,to=2 -monitor stdio
QEMU 2.8.1.1 monitor - type 'help' for more information
(qemu) info vnc
Server:
     address: 0.0.0.0:5902

Comment 2 Gerd Hoffmann 2017-04-27 06:54:55 UTC
kraxel@nilsson ~/Downloads/patches# netstat -tnap | grep qemu
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 0.0.0.0:5910            0.0.0.0:*               LISTEN      8774/qemu-system-x8 
tcp6       0      0 :::5911                 :::*                    LISTEN      8774/qemu-system-x8 

qemu appears to use two ports now, one for ipv4, one for ipv6.

Comment 3 Guo, Zhiyi 2017-04-27 08:00:02 UTC
(In reply to Gerd Hoffmann from comment #2)
> kraxel@nilsson ~/Downloads/patches# netstat -tnap | grep qemu
> (Not all processes could be identified, non-owned process info
>  will not be shown, you would have to be root to see it all.)
> tcp        0      0 0.0.0.0:5910            0.0.0.0:*               LISTEN  
> 8774/qemu-system-x8 
> tcp6       0      0 :::5911                 :::*                    LISTEN  
> 8774/qemu-system-x8 
> 
> qemu appears to use two ports now, one for ipv4, one for ipv6.

Hi Gerd,

   I want to verify Bug 1038963 - [RFE] qemu can't listen on both IPv6 and IPv4 localhost for VNC  but qemu only bind to ipv4 address with my cli:
# /usr/libexec/qemu-kvm -vnc :0 -monitor stdio
QEMU 2.9.0 monitor - type 'help' for more information
(qemu) info vnc
Server:
     address: 0.0.0.0:5900
        auth: none
Client: none
(qemu) 

# netstat -tnap | grep qemu
tcp        0      0 0.0.0.0:5900            0.0.0.0:*               LISTEN      5840/qemu-kvm       

Could you give us a reference how to verify that bug?

BR/
Guo, Zhiyi

Comment 4 Daniel Berrangé 2017-04-27 08:41:59 UTC
(In reply to Gerd Hoffmann from comment #2)
> kraxel@nilsson ~/Downloads/patches# netstat -tnap | grep qemu
> (Not all processes could be identified, non-owned process info
>  will not be shown, you would have to be root to see it all.)
> tcp        0      0 0.0.0.0:5910            0.0.0.0:*               LISTEN  
> 8774/qemu-system-x8 
> tcp6       0      0 :::5911                 :::*                    LISTEN  
> 8774/qemu-system-x8 
> 
> qemu appears to use two ports now, one for ipv4, one for ipv6.

That would be a bug - we should only ever use a single port, even if using 'to='. If we do end up having multiple listening sockets, then we must ensure they all use the same port as the first socket. I guess this is a regression from the listener rewrite I did

Comment 5 Daniel Berrangé 2017-04-27 09:30:08 UTC
So when you use ':0', the host name is "" and getaddrinfo resolves that to "0.0.0.0" and "::", in that order.

We listen on "0.0.0.0" first and succeed.

The old code stopped there and didn't even bother trying to listen on "::".

The new code tries "::" and because we have IPV6_V6ONLY=0, it fails to listen on "::" since it clashes with 0.0.0.0. Because to=2 is set, we try "::" again on the next port.

What we really need here is for IPV6_V6ONLY to be set to 1, so that we when try "::", we don't clash with "0.0.0.0".

The problem with changing IPV6_V6ONLY to 1 though, is that then breaks people who have explicitly set '::' as their listen address.

Comment 6 Ademar Reis 2017-04-28 16:05:29 UTC
(In reply to Daniel Berrange from comment #4)
> (In reply to Gerd Hoffmann from comment #2)
> > kraxel@nilsson ~/Downloads/patches# netstat -tnap | grep qemu
> > (Not all processes could be identified, non-owned process info
> >  will not be shown, you would have to be root to see it all.)
> > tcp        0      0 0.0.0.0:5910            0.0.0.0:*               LISTEN  
> > 8774/qemu-system-x8 
> > tcp6       0      0 :::5911                 :::*                    LISTEN  
> > 8774/qemu-system-x8 
> > 
> > qemu appears to use two ports now, one for ipv4, one for ipv6.
> 
> That would be a bug - we should only ever use a single port, even if using
> 'to='. If we do end up having multiple listening sockets, then we must
> ensure they all use the same port as the first socket. I guess this is a
> regression from the listener rewrite I did

Would you be able to fix it and submit the backport?

Comment 7 Daniel Berrangé 2017-05-02 08:10:09 UTC
Possible patch provided upstream

https://lists.gnu.org/archive/html/qemu-devel/2017-04/msg05659.html

Comment 8 Ademar Reis 2017-05-16 14:42:27 UTC
(In reply to Daniel Berrange from comment #7)
> Possible patch provided upstream
> 
> https://lists.gnu.org/archive/html/qemu-devel/2017-04/msg05659.html

I see there was some discussion upstream, but looks like it should be merged. v2?

Comment 10 Daniel Berrangé 2017-05-16 14:46:19 UTC
(In reply to Ademar Reis from comment #8)
> (In reply to Daniel Berrange from comment #7)
> > Possible patch provided upstream
> > 
> > https://lists.gnu.org/archive/html/qemu-devel/2017-04/msg05659.html
> 
> I see there was some discussion upstream, but looks like it should be
> merged. v2?

Yeah, i'll be sending a v2

Comment 11 Daniel Berrangé 2017-05-22 12:43:37 UTC
The fix for this has triggered a broader cleanup. The specific fix for this bug is in the 1st patch here

https://lists.gnu.org/archive/html/qemu-devel/2017-05/msg04706.html

I'll file a separate bugs to track the other issues described in that series.

Comment 16 Miroslav Rezanina 2017-06-23 06:12:51 UTC
Fix included in qemu-kvm-rhev-2.9.0-14.el7

Comment 18 Guo, Zhiyi 2017-06-26 02:45:51 UTC
Test against qemu-kvm-rhev-2.9.0-14.el7.x86_64

# /usr/libexec/qemu-kvm -vnc :0

# netstat -pant | grep 5900
tcp        0      0 0.0.0.0:5900            0.0.0.0:*               LISTEN      30179/qemu-kvm      
tcp6       0      0 :::5900                 :::*                    LISTEN      30179/qemu-kvm      
   
vnc can listen on both ipv4 and ipv6

Launch 1st qemu process:
# /usr/libexec/qemu-kvm -vnc :0,to=1 -monitor stdio
QEMU 2.9.0 monitor - type 'help' for more information
(qemu) VNC server running on 0.0.0.0:5900

Keep 1st qemu process alive and luanch 2nd qemu process:
# /usr/libexec/qemu-kvm -vnc :0,to=1 -monitor stdio
QEMU 2.9.0 monitor - type 'help' for more information
(qemu) VNC server running on 0.0.0.0:5901

Keep 1st and 2nd qemu process alive and launch 3rd qemu process:
# /usr/libexec/qemu-kvm -vnc :0,to=1
qemu-kvm: -vnc :0,to=1: Failed to start VNC server: Failed to bind socket: Address already in use

Both 5900 and 5901 can be connected with remote-viewer
So the bug is fixed

Comment 19 Guo, Zhiyi 2017-06-26 02:46:31 UTC
Verified per comment 18

Comment 21 errata-xmlrpc 2017-08-02 04:35:59 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392