Are you by any chance located in China, behind the not-so-great firewall?

It shouldn't be possible to end up in a situation where _sslobj is None. We need a description of how to reproduce it ... or someone who can reproduce it and has the skills to debug it.

Yes, I am in Mainland China. That should be the very problem.

If you know some python then please try to debug it.
Edit /usr/lib*/python2.7/ssl.py and try to figure out why _sslobj is None. A stacktrace showing where it is set to None would be helpful.

I peak at mercurial/sslutil.py of version 2.0.2. At line 107, it may be better first test the connection through sock.cipher() before running sock.getpeercert .

Yes, that or some other workaround could have given a custom error message.

There is however no indication in the python documentation that cipher should be checked first or that getpeercert should fail. It thus looks a lot like a Python bug, but before I file a bug report or make a workaround in Mercurial I would like to understand in which situations this can occur and be sure that it isn't a bug in Mercurials use of ssl.

Please try to figure out when and why _sslobj silently is set to None.

At ssl.py SSLSocket.__init__:

        try:
            socket.getpeername(self)
        except socket_error, e:
            if e.errno != errno.ENOTCONN:
                raise
            # no, no connection yet                                                       
            self._connected = False
            self._sslobj = None # !!!! _sslobj was set to None

And e.errno is 107.

How about asking the question at the Python community? I may not be able to dig into the C implementation. I am not familiar with Python C API.

Thanks.

I haven't seen the problem myself, and until now I haven't seen enough reliable evidence to know what to say and ask for. But we are getting closer. You are welcome to go ahead now and investigate and report it with a minimal test case.

To help us understand (more of) the exact conditions where it fails, can you please show the output of 
  tcpdump -n port 443
of a failing clone?

Created attachment 550792 [details]
output of "tcpdump -n port 443" during a blocked hg through https

Thanks. We are getting closer.

Do the dump show several 'hg clone' attempts?
Do each attempt hang for 30 seconds before it fails?
Did all attempts fail the same way, or was it only the last one that failed with the crash we are discussing?

Depending on answers to these questions I assume that the connection attempts simply was blocked in different ways by the wall. Could you try to specify different IP addresses of code.google.com explicitly and see if they simply are blocked in different ways?

I wonder why the first connections stopped after 30 seconds. Can you try
  tcpdump -n port 443 or icmp
and see if something reports failure back over ICMP?

I assume that the following always fails nicely with a 'connection reset':

import socket, ssl
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
ssl_sock = ssl.wrap_socket(s)
ssl_sock.connect(('code.google.com', 443))
ssl_sock.getpeercert(True)

while this can fail with the NoneType AttributeError:

import socket, ssl
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('code.google.com', 443))
ssl_sock = ssl.wrap_socket(s)
ssl_sock.getpeercert(True)

?

(In reply to comment #9)
> Do the dump show several 'hg clone' attempts?
How to check this?

> Do each attempt hang for 30 seconds before it fails?
There are 30 seconds between last failure and an attempt to another IP address. Failures are incurred by checksum incorrect.

> Did all attempts fail the same way, or was it only the last one that failed
> with the crash we are discussing?
I can summarize the three kinds of situations:

 1) Connection timeout and hg throws NoneType. 'hg clone' tries several IP addresses of the domain and the transmissions failed mostly with checksum incorrect.

 2) Connection reset and hg fails quickly and shows the right error.

 3) Connection reset and hg fails quickly and throws NoneType.

> 
> Depending on answers to these questions I assume that the connection attempts
> simply was blocked in different ways by the wall. Could you try to specify
> different IP addresses of code.google.com explicitly and see if they simply are
> blocked in different ways?
$ hg clone https://74.125.71.100/p/lonote/
abort: error: Connection reset by peer

This is just the second situation. And the other two also occur with explicit IP addresses.

> 
> I wonder why the first connections stopped after 30 seconds. Can you try
>   tcpdump -n port 443 or icmp
> and see if something reports failure back over ICMP?
There seems no ICMP report.


> I assume that the following always fails nicely with a 'connection reset':
> 
> import socket, ssl
> s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
> ssl_sock = ssl.wrap_socket(s)
> ssl_sock.connect(('code.google.com', 443))
> ssl_sock.getpeercert(True)
> 
> while this can fail with the NoneType AttributeError:
> 
> import socket, ssl
> s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
> s.connect(('code.google.com', 443))
> ssl_sock = ssl.wrap_socket(s)
> ssl_sock.getpeercert(True)
> 
> ?
> 
Result of the first script:

$ python expect_Reset.py 
Traceback (most recent call last):
  File "expect_Reset.py", line 8, in <module>
    ssl_sock.connect(('code.google.com', 443))
  File "/usr/lib64/python2.7/ssl.py", line 322, in connect
    self._real_connect(addr, False)
  File "/usr/lib64/python2.7/ssl.py", line 315, in _real_connect
    raise e
socket.error: [Errno 110] Connection timed out

Result of the second script:

$ python expect_NoneType.py 
Traceback (most recent call last):
  File "expect_NoneType.py", line 6, in <module>
    s.connect(('code.google.com', 443))
  File "/usr/lib64/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
socket.error: [Errno 110] Connection timed out

Since the Firewall is not static, the result may differ like the three situations.

(In reply to comment #10)
> (In reply to comment #9)
> > Do the dump show several 'hg clone' attempts?
> How to check this?

Did you issue one or several 'hg clone' commands while tcpdump was running? 

> > Do each attempt hang for 30 seconds before it fails?
> There are 30 seconds between last failure and an attempt to another IP address.

'hg clone' is automatically failing over to the next IP address?

> Failures are incurred by checksum incorrect.

What observation leads to that conclusion? You don't receive any response at all, so how can the checksum be incorrect?

> > Did all attempts fail the same way, or was it only the last one that failed
> > with the crash we are discussing?
> I can summarize the three kinds of situations:
> 
>  1) Connection timeout and hg throws NoneType. 'hg clone' tries several IP
> addresses of the domain and the transmissions failed mostly with checksum
> incorrect.

One 'hg clone https://...' invocation will retry several IP addresses? That is very surprising and not indicated by source or documentation and I can't reproduce that.

Are you using plain Mercurial without any wrappers or modifications or extensions?

>  2) Connection reset and hg fails quickly and shows the right error.
> 
>  3) Connection reset and hg fails quickly and throws NoneType.

That is very surprising and not the behaviour I would expect from looking at the source or documentation and I can't reproduce that.

Can you correlate the different failure modes with different tcpdumps?

> > Depending on answers to these questions I assume that the connection attempts
> > simply was blocked in different ways by the wall. Could you try to specify
> > different IP addresses of code.google.com explicitly and see if they simply are
> > blocked in different ways?
> $ hg clone https://74.125.71.100/p/lonote/
> abort: error: Connection reset by peer
> 
> This is just the second situation. And the other two also occur with explicit
> IP addresses.

Is the same IP address able to fail in different ways?

> > I wonder why the first connections stopped after 30 seconds. Can you try
> >   tcpdump -n port 443 or icmp
> > and see if something reports failure back over ICMP?
> There seems no ICMP report.

So what makes the connection fail after 30 seconds? When I configure my router to reproduce the 'no response at all' scenario it keeps retrying much longer. Do you have any special configuration on your machine? Or is it plain f16?

> $ python expect_Reset.py 
> Traceback (most recent call last):
>   File "expect_Reset.py", line 8, in <module>
>     ssl_sock.connect(('code.google.com', 443))
>   File "/usr/lib64/python2.7/ssl.py", line 322, in connect
>     self._real_connect(addr, False)
>   File "/usr/lib64/python2.7/ssl.py", line 315, in _real_connect
>     raise e
> socket.error: [Errno 110] Connection timed out
> 
> Result of the second script:
> 
> $ python expect_NoneType.py 
> Traceback (most recent call last):
>   File "expect_NoneType.py", line 6, in <module>
>     s.connect(('code.google.com', 443))
>   File "/usr/lib64/python2.7/socket.py", line 224, in meth
>     return getattr(self._sock,name)(*args)
> socket.error: [Errno 110] Connection timed out
> 
> Since the Firewall is not static, the result may differ like the three
> situations.

Hmm. That seems inconsistent and contradicts my theory of what the problem is. I thought I had found a way to reproduce the problem here ... bat apprently not.

Are these behaviours as consistently reproducible as the 'hg clone' failure?

How long time does it take before they fail?

(In reply to comment #11)
> 'hg clone' is automatically failing over to the next IP address?

Ok, yes it is.

> One 'hg clone https://...' invocation will retry several IP addresses? That is
> very surprising and not indicated by source or documentation and I can't
> reproduce that.

Ok, Pythons socket.create_connection documention has 'room for improvement'.

Most of comment #11 is no longer relevant, but:

Is the behaviour of the test scripts 100% consistent?

Can you do 
  for i in $(seq 100); do echo $i; python expect_Reset.py; done
and
  for i in $(seq 100); do echo $i; python expect_None.py; done

and see if you ever get anything else than timeout?

A connection reset result of the first script:

Traceback (most recent call last):
  File "expect_Reset.py", line 8, in <module>
    ssl_sock.connect(('code.google.com', 443))
  File "/usr/lib64/python2.7/ssl.py", line 322, in connect
    self._real_connect(addr, False)
  File "/usr/lib64/python2.7/ssl.py", line 315, in _real_connect
    raise e
socket.error: [Errno 104] Connection reset by peer

A connection reset result of the second script:
Traceback (most recent call last):
  File "expect_NoneType.py", line 7, in <module>
    ssl_sock = ssl.wrap_socket(s)
  File "/usr/lib64/python2.7/ssl.py", line 372, in wrap_socket
    ciphers=ciphers)
  File "/usr/lib64/python2.7/ssl.py", line 134, in __init__
    self.do_handshake()
  File "/usr/lib64/python2.7/ssl.py", line 296, in do_handshake
    self._sslobj.do_handshake()
socket.error: [Errno 104] Connection reset by peer


There are also many timeouts. No other kind of result was shown during the 100 tries.

That was fast. I assume that means that they all fail instantaneously? Does that mean that a tcpdump today shows immediate responses from the firewall? But 'hg clone' still hangs for a long time? 

(Logging/debugging of the hanging 'hg clone' can be improved temporarily with
--- /usr/lib64/python2.7/socket.py.org	2012-01-06 14:46:01.128099111 +0100
+++ /usr/lib64/python2.7/socket.py	2012-01-06 14:13:14.732211105 +0100
@@ -551,6 +551,7 @@
     host, port = address
     err = None
     for res in getaddrinfo(host, port, 0, SOCK_STREAM):
+        print res
         af, socktype, proto, canonname, sa = res
         sock = None
         try:
@@ -564,6 +565,7 @@
 
         except error as _:
             err = _
+            print err
             if sock is not None:
                 sock.close()
)


Do you see the problem - and slow connects - with 100 times:

import socket, ssl
s = socket.create_connection(('code.google.com', 443))
ssl_sock = ssl.wrap_socket(s)
ssl_sock.getpeercert(True)

- which essentially is what Mercurial do. ?

Created attachment 551168 [details]
Output of the script in #c14

(In reply to comment #15)

Ok, that shows that something like 39% of the connect attempts times out after 60 seconds (because they don't get any response at all, like the first 3 attempts in your first tcpdump). The other 61% gets a 'connection reset' (like in the 4th attempt in your first tcpdump) (which stops the create_connection looping). But we see the reset while doing the handshake, so the getpeername that failed for Mercurial must have succeeded. I guess that is because the RESET from the peer haven't been received yet. Can you try with an extra delay:

import socket, ssl, time
s = socket.create_connection(('code.google.com', 443))
time.sleep(1)
ssl_sock = ssl.wrap_socket(s)
ssl_sock.getpeercert(True)


Do 'hg clone' with the debug annotation in ssl.py also show this pattern with something like 39% timeout+failover?

With the delay, the output all turns to something like:

(2, 1, 6, '', ('74.125.71.101', 443))
[Errno 110] Connection timed out
(2, 1, 6, '', ('74.125.71.100', 443))
[Errno 110] Connection timed out
(2, 1, 6, '', ('74.125.71.102', 443))
[Errno 110] Connection timed out
(2, 1, 6, '', ('74.125.71.139', 443))
Traceback (most recent call last):
  File "socket-create_connection.py", line 8, in <module>
    ssl_sock.getpeercert(True)
  File "/usr/lib64/python2.7/ssl.py", line 172, in getpeercert
    return self._sslobj.peer_certificate(binary_form)
AttributeError: 'NoneType' object has no attribute 'peer_certificate'

real    3m10.466s
user    0m0.040s
sys     0m0.009s


> Do 'hg clone' with the debug annotation in ssl.py also show this pattern with
> something like 39% timeout+failover?
hg should be in the same pattern.

Excellent, thank you. I think that makes the problem and its impact fully understood.

The crash will be fixed in the next Mercurial release Februar 1st which will be included in Fedora 17.

It will obviously still not be possible to bypass the wall. The fix will thus be purely cosmetic and I doubt it will be back-ported to Fedora 16.

I am happy to test it before final release.

It has been fixed in upstream Mercurial in http://selenic.com/repo/hg/rev/0cc4ad757c77 - and improved further in later changesets.

You can test it with something like:
sudo yum-builddep mercurial
hg clone http://selenic.com/hg
cd hg
make local
./hg clone https://... /tmp/...

No exception is thrown now.

Thanks for helping with the debugging and testing!

hg clone somethings from googlecode.com

Package: mercurial-1.9.3-1.fc16
Architecture: i686
OS Release: Fedora release 16 (Verne)