Bug 1472759

Summary: RFE: Report AF_VSOCK socket status in ss(8)
Product: Red Hat Enterprise Linux 7 Reporter: Stefan Hajnoczi <stefanha>
Component: iprouteAssignee: Stefano Brivio <sbrivio>
Status: CLOSED ERRATA QA Contact: Jaroslav Aster <jaster>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.5CC: aloughla, areis, atragler, boyang, jaster, ksrot, ldu, leiwang, psutter, qe-baseos-security, sbrivio, stefanha, yacao
Target Milestone: rcKeywords: FutureFeature
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: iproute-4.11.0-11.el7 Doc Type: Enhancement
Doc Text:
Feature: It is now possible to list AF_VSOCK sockets using 'ss' tool. Reason: AF_VSOCK sockets can be used by KVM and VMware to communicate between the hypervisor and virtual machines. ss(8) therefore could not report detailed information on listen sockets or established connections. Result: ss tool now allows to observe system status regarding open AF_VSOCK sockets.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2018-04-10 14:31:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1470203    
Bug Blocks:    

Description Stefan Hajnoczi 2017-07-19 11:23:53 UTC 
Description of problem:

There is currently no way to view the socket status for AF_VSOCK sockets.  AF_VSOCK sockets can be used by KVM and VMware to communicate between the hypervisor and virtual machines.  ss(8) therefore cannot report detailed information on listen sockets or established connections.

Users, developers, and system administrators expect to be able to see sockets used by programs.  This is helpful for troubleshooting and debugging.

I'm filing this BZ as a feature request and I expect to send patches upstream myself to implement this feature.

Steps to Reproduce:
1. Create a listen socket on the host with "nc-vsock -l 1234" (see http://github.com/stefanha/nc-vsock/).
2. Check that the listen socket is reported on the host: "ss"
3. Connect to the socket from the guest with "nc-vsock 2 1234"
4. Check that the established connection is reported in the guest: "ss"
5. Check that the established connection is reported on the host: "ss"
6. End the connection inside the guest with Ctrl+D and check that it is no longer listed by "ss"
7. Check that connection is no longer listed on the host with "ss" and the listen socket should also be gone.
Comment 2 Stefan Hajnoczi 2017-10-12 08:54:31 UTC 
The patches have been merged upstream in the iproute2.git net-next branch:
https://git.kernel.org/pub/scm/linux/kernel/git/shemminger/iproute2.git/commit/?h=net-next&id=c759116a0b2b6da8df9687b0a40ac69050132c77
Comment 3 Stefano Brivio 2017-10-12 08:56:29 UTC 
(In reply to Stefan Hajnoczi from comment #2)
> The patches have been merged upstream in the iproute2.git net-next branch:
> https://git.kernel.org/pub/scm/linux/kernel/git/shemminger/iproute2.git/
> commit/?h=net-next&id=c759116a0b2b6da8df9687b0a40ac69050132c77

ACK, thanks for reporting this.
Comment 7 Phil Sutter 2017-11-11 12:04:21 UTC 
Anita, could you please provide pm_ack+?
Comment 9 Jaroslav Aster 2017-12-05 14:24:36 UTC 
Hi,

I'm trying to test this bug, but it does not work. Do I do something wrong?

[# ./nc-vsock -l 1234 &
[1] 10020

# jobs
[1]+  Running                 ./nc-vsock -l 1234 &

# ss -a -n --vsock
Netid State      Recv-Q Send-Q                                                           Local Address:Port                                                                          Peer Address:Port     

I expect to see one vsock in listen state, but I don't see nothing :-).
Comment 10 Ademar Reis 2017-12-05 17:05:49 UTC 
(In reply to Jaroslav Aster from comment #9)
> Hi,
> 
> I'm trying to test this bug, but it does not work. Do I do something wrong?
> 
> [# ./nc-vsock -l 1234 &
> [1] 10020
> 
> # jobs
> [1]+  Running                 ./nc-vsock -l 1234 &
> 
> # ss -a -n --vsock
> Netid State      Recv-Q Send-Q                                              
> Local Address:Port                                                          
> Peer Address:Port     
> 
> I expect to see one vsock in listen state, but I don't see nothing :-).

Stefan might be able to help as well.
Comment 11 Stefano Brivio 2017-12-07 16:56:01 UTC 
Hi Jaroslav,

(In reply to Jaroslav Aster from comment #9)
> Hi,
> 
> I'm trying to test this bug, but it does not work. Do I do something wrong?
> 
> [# ./nc-vsock -l 1234 &
> [1] 10020
> 
> # jobs
> [1]+  Running                 ./nc-vsock -l 1234 &
> 
> # ss -a -n --vsock
> Netid State      Recv-Q Send-Q                                              
> Local Address:Port                                                          
> Peer Address:Port     
> 
> I expect to see one vsock in listen state, but I don't see nothing :-).

Sorry for the delay here. Which kernel version are you using? Please note that this feature needs the following patches:

        http://patchwork.usersys.redhat.com/patch/190081/
        http://patchwork.usersys.redhat.com/patch/190080/
        http://patchwork.usersys.redhat.com/patch/191738/
        http://patchwork.usersys.redhat.com/patch/190082/
        http://patchwork.usersys.redhat.com/patch/190078/

in order to work properly. They are available only starting from kernel-3.10.0-773.el7.
Comment 12 Jaroslav Aster 2017-12-08 13:10:21 UTC 
Hi,

I tested it on 799.

# ./nc-vsock -l 1234 &
[1] 9815
# rpm -q iproute
iproute-4.11.0-11.el7.x86_64
# ss -a -n --vsock
Netid State      Recv-Q Send-Q                                                           Local Address:Port                                                                          Peer Address:Port              
# uname -r
3.10.0-799.el7.x86_64
Comment 13 Stefan Hajnoczi 2017-12-08 16:08:19 UTC 
(In reply to Jaroslav Aster from comment #12)
> Hi,
> 
> I tested it on 799.
> 
> # ./nc-vsock -l 1234 &
> [1] 9815
> # rpm -q iproute
> iproute-4.11.0-11.el7.x86_64
> # ss -a -n --vsock
> Netid State      Recv-Q Send-Q                                              
> Local Address:Port                                                          
> Peer Address:Port              
> # uname -r
> 3.10.0-799.el7.x86_64

Are you sure the vsock_diag.ko kernel module is available?

Try:

  # strace -f ss -a -n --vsock

Check the SOCK_DIAG_BY_FAMILY sendto() system call and the recvmsg() reply:

sendto(3, {{len=40, type=SOCK_DIAG_BY_FAMILY, flags=NLM_F_REQUEST|NLM_F_DUMP, seq=123456, pid=0}, {family=AF_VSOCK, "\x00\x00\x00\x80\x04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"}}, 40, 0, NULL, 0) = 40
recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base={{len=60, type=NLMSG_ERROR, flags=0, seq=123456, pid=14507}, {error=-ENOENT, msg={{len=40, type=SOCK_DIAG_BY_FAMILY, flags=NLM_F_REQUEST|NLM_F_DUMP, seq=123456, pid=0}, {family=AF_VSOCK, "\x00\x00\x00\x80\x04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"}}}}, iov_len=32768}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 60

In this example NLMSG_ERROR and error=-ENOENT indicate that the vsock_diag.ko module is not available.
Comment 14 Jaroslav Aster 2017-12-08 16:46:18 UTC 
Hi,

on my system, there is no kernel module vsock_diag.ko available.

# find /lib/modules/3.10.0-799.el7.x86_64 -type f |grep vsock
/lib/modules/3.10.0-799.el7.x86_64/kernel/drivers/net/vsockmon.ko.xz
/lib/modules/3.10.0-799.el7.x86_64/kernel/drivers/vhost/vhost_vsock.ko.xz
/lib/modules/3.10.0-799.el7.x86_64/kernel/net/vmw_vsock/hv_sock.ko.xz
/lib/modules/3.10.0-799.el7.x86_64/kernel/net/vmw_vsock/vmw_vsock_virtio_transport.ko.xz
/lib/modules/3.10.0-799.el7.x86_64/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko.xz
/lib/modules/3.10.0-799.el7.x86_64/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko.xz
/lib/modules/3.10.0-799.el7.x86_64/kernel/net/vmw_vsock/vsock.ko.xz

# lsmod|grep vsock
vmw_vsock_vmci_transport    30577  1 
vmw_vmci               67081  1 vmw_vsock_vmci_transport
vsock                  36452  3 vmw_vsock_vmci_transport

The only sendto, I can see in the strace output, is this:

# strace -f ss -a -n --vsock |& grep sendto
sendto(3, "(\0\0\0\24\0\1\3@\342\1\0\0\0\0\0(\0\0\0\377\17\0\0\0\0\0\0\0\0\0\0"..., 40, 0, NULL, 0) = 40
Comment 29 errata-xmlrpc 2018-04-10 14:31:17 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:0815