Bug 983630

Summary: [vdsm] engine address does not appear in vdsm.log when SSL connection problem occurs
Product: Red Hat Enterprise Virtualization Manager Reporter: Elad <ebenahar>
Component: vdsmAssignee: Juan Hernández <juan.hernandez>
Status: CLOSED ERRATA QA Contact: Artyom <alukiano>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 3.2.0CC: acathrow, bazulay, danken, ebenahar, iheim, jkt, juan.hernandez, lpeer, pstehlik, yeylon
Target Milestone: ---Keywords: Reopened
Target Release: 3.3.0   
Hardware: x86_64   
OS: Unspecified   
Whiteboard: infra
Fixed In Version: is16 Doc Type: Bug Fix
Doc Text:
The client IP address used to be logged when SSL errors occurred, but this stopped working with the introduction of cached SSL sessions. This update modifies the SSLServerSocket class that was introduced to implement cached SSL sessions. Now when an SSL error is detected while attempting a connection, the IP address is added to the message of the exception and logged.
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-01-21 16:27:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Infra RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Elad 2013-07-11 15:05:45 UTC
Description of problem:
engine address is not shown in vdsm.log when we get this message:

BindingXMLRPC::ERROR::2013-07-10 21:01:09,398::BindingXMLRPC::72::vds::(threaded_start) xml-rpc handler exception
Traceback (most recent call last):
  File "/usr/share/vdsm/BindingXMLRPC.py", line 68, in threaded_start
    self.server.handle_request()
  File "/usr/lib64/python2.6/SocketServer.py", line 268, in handle_request
    self._handle_request_noblock()
  File "/usr/lib64/python2.6/SocketServer.py", line 278, in _handle_request_noblock
    request, client_address = self.get_request()
  File "/usr/lib64/python2.6/SocketServer.py", line 446, in get_request
    return self.socket.accept()
  File "/usr/lib64/python2.6/site-packages/vdsm/SecureXMLRPCServer.py", line 116, in accept
    client, address = self.connection.accept()
  File "/usr/lib64/python2.6/site-packages/M2Crypto/SSL/Connection.py", line 167, in accept
    ssl.accept_ssl()
  File "/usr/lib64/python2.6/site-packages/M2Crypto/SSL/Connection.py", line 156, in accept_ssl
    return m2.ssl_accept(self.ssl, self._timeout)
SSLError: sslv3 alert certificate unknown



Version-Release number of selected component (if applicable):
vdsm-4.10.2-23.0.el6ev.x86_64

Expected results:
engine address should appear in vdsm.log when SSL connection problem occurs

Comment 1 Juan Hernández 2013-07-15 21:49:46 UTC
Elad, can you tell me how are you generating that "sslv3 alert certificate unknown" message? I can't reproduce it. I can reproduce other SSL alerts, but not this one, and all the ones that I reproduce are caught later, after the accept method, and thus are processed correctly.

Anyhow, I think that we may need to modify the accept method of SSLServerSocket to something like this:

    def accept(self):
        # First we need to call the accept method of the raw
        # socket and build a new SSL connection object with the
        # results:
        client, address = self.connection.socket.accept()
        client = SSL.Connection(self.context, client)
        client.addr = address

        # Now perform the SSL setup and log errors including the
        # address of the remote client:
        try:
            client.setup_ssl()
            client.set_accept_state()
        except:
            logging.error('client %s', address, exc_info=True)
            raise

        # Finally wrap the connection so that it mimics a
        # socket as the XMLRPC server expects:
        client = SSLSocket(client)

        return client, address

Comment 2 Elad 2013-09-10 06:47:20 UTC
(In reply to Juan Hernández from comment #1)
> Elad, can you tell me how are you generating that "sslv3 alert certificate
> unknown" message? I can't reproduce it. I can reproduce other SSL alerts,
> but not this one, and all the ones that I reproduce are caught later, after
> the accept method, and thus are processed correctly.
> 
> Anyhow, I think that we may need to modify the accept method of
> SSLServerSocket to something like this:
> 
>     def accept(self):
>         # First we need to call the accept method of the raw
>         # socket and build a new SSL connection object with the
>         # results:
>         client, address = self.connection.socket.accept()
>         client = SSL.Connection(self.context, client)
>         client.addr = address
> 
>         # Now perform the SSL setup and log errors including the
>         # address of the remote client:
>         try:
>             client.setup_ssl()
>             client.set_accept_state()
>         except:
>             logging.error('client %s', address, exc_info=True)
>             raise
> 
>         # Finally wrap the connection so that it mimics a
>         # socket as the XMLRPC server expects:
>         client = SSLSocket(client)
> 
>         return client, address

It happens when we have 2 engines that are connected by ssl to the same host

Comment 3 Barak 2013-09-16 13:42:34 UTC
according to comment #2 this is not a bug
Moving to CLOSE NOTABUG

Comment 4 Elad 2013-09-16 14:14:34 UTC
(In reply to Barak from comment #3)
> according to comment #2 this is not a bug
> Moving to CLOSE NOTABUG

IMO, having the relevant engine address is necessary in that error.

Comment 5 Dan Kenigsberg 2013-09-16 15:31:41 UTC
I'd like to solve this recent regression: if there's a DoS attack on a vdsm (malevolent or unintentional), we'd like to log the IP of the attacker.

Comment 6 Artyom 2013-10-13 05:27:11 UTC
Verified on is18
In the end of error message appear message
SSLError: sslv3 alert certificate unknown, client 10.35.161.82

Comment 7 Charlie 2013-11-28 00:31:12 UTC
This bug is currently attached to errata RHBA-2013:15291. If this change is not to be documented in the text for this errata please either remove it from the errata, set the requires_doc_text flag to 
minus (-), or leave a "Doc Text" value of "--no tech note required" if you do not have permission to alter the flag.

Otherwise to aid in the development of relevant and accurate release documentation, please fill out the "Doc Text" field above with these four (4) pieces of information:

* Cause: What actions or circumstances cause this bug to present.
* Consequence: What happens when the bug presents.
* Fix: What was done to fix the bug.
* Result: What now happens when the actions or circumstances above occur. (NB: this is not the same as 'the bug doesn't present anymore')

Once filled out, please set the "Doc Type" field to the appropriate value for the type of change made and submit your edits to the bug.

For further details on the Cause, Consequence, Fix, Result format please refer to:

https://bugzilla.redhat.com/page.cgi?id=fields.html#cf_release_notes 

Thanks in advance.

Comment 8 errata-xmlrpc 2014-01-21 16:27:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-0040.html