Created attachment 1151115 [details]
working cert

Created attachment 1151116 [details]
working key

Created attachment 1151118 [details]
working ca cert

Created attachment 1151119 [details]
not working cert

Created attachment 1151121 [details]
not working key

Created attachment 1151122 [details]
not working ca cert

What's the version of libsoup in use here?  Or more generally what is the system hosting ostree?  Is it e.g. RHEL 7.2, is it Fedora 24, etc?

And just to sanity check, can you access the Akamai CDN with this copy of ostree using entitled certs?

Colin,

It is using libsoup-2.48.1-3.el7.x86_64 and it is a centos 7.1 machine

Neither of these certs will work with the cdn as they are generated by the Satellite.

I would like to clarify a few points:

- the satellite is generating a 'copy' of the cdn
- the satellite is generating its own entitlement certs (attached as "working") which work fine with ostree   
- the satellite is generating another ueber cert (attached as "not working") that does not work with ostree, but works with curl, yum, etc....

FWIW, Tried this on updated f23 box and got same error:

[jortel@f23d ostree]$ ostree --repo=. pull pulp


error: Server required TLS certificate

____________________

Installed:

ostree-2015.11-2.fc23.x86_64
libsoup-2.52.2-1.fc23.x86_64

I am wondering what the status of this bug is and if any more information is needed. Is there any more information we can provide? Any logging we can turn on for more debugging?

Sorry, meant to update it.  I've reproduced the problem, I suspect it's something going wrong inside the glib-networking gnutls integration.

There is the OSTREE_DEBUG_HTTP environment variable but it doesn't help with this.

Okay, so a very important detail here:

 - curl uses NSS (in Fedora/RHEL)
 - ostree uses glib-networking which currently uses GnuTLS (though there are non-upstream patches for OpenSSL)

Apparently coercing NSS to emit useful debuginfo requires recompiling it, which I tried but still didn't get it to work. ( https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/Reference/NSS_environment_variables )  Will look at that more in a bit.

In contrast, GnuTLS offers GNUTLS_DEBUG_LEVEL: https://gnutls.org/manual/html_node/Debugging-and-auditing.html
which is pretty much exactly what we need.

The relevant failure seems to be:

...
gnutls[4]: HSK[0x7f3178315d00]: SERVER HELLO DONE (14) was received. Length 0[0], frag offset 0, frag length: 1, sequence: 0
gnutls[4]: HSK[0x7f3178315d00]: CERTIFICATE was queued [1759 bytes]
gnutls[4]: HSK[0x7f3178315d00]: CLIENT KEY EXCHANGE was queued [70 bytes]
gnutls[4]: HSK[0x7f3178315d00]: CERTIFICATE VERIFY was queued [264 bytes]
gnutls[4]: REC[0x7f3178315d00]: Sent ChangeCipherSpec
...
gnutls[4]: HSK[0x7f3178315d00]: Cipher Suite: ECDHE_RSA_AES_256_GCM_SHA384
gnutls[4]: HSK[0x7f3178315d00]: Initializing internal [write] cipher sessions
gnutls[4]: HSK[0x7f3178315d00]: recording tls-unique CB (send)
gnutls[4]: HSK[0x7f3178315d00]: FINISHED was queued [16 bytes]
...
gnutls[5]: REC[0x7fea901ff330]: SSL 3.3 Alert packet received. Epoch 0, length: 2
gnutls[5]: REC[0x7fea901ff330]: Expected Packet Handshake(22)
...
gnutls[5]: REC[0x7f3178315d00]: Alert[2|51] - Decrypt error - was received


For a successful, run, the last line looks like:

gnutls[5]: REC[0x7feba424fed0]: Received Packet Handshake(22) with length: 48

I do notice at least with Akamai, we negotiate:

gnutls[4]: HSK[0x7feba424fed0]: Cipher Suite: RSA_AES_256_CBC_SHA1

But AFAIK both should be fine, certainly GnuTLS knows about it:

# gnutls-cli -l |grep  ECDHE_RSA_AES_256_GCM_SHA384
TLS_ECDHE_RSA_AES_256_GCM_SHA384                  	0xc0, 0x30	TLS1.2
#

It doesn't seem to be related to the cipher suite, if I e.g. use G_TLS_GNUTLS_PRIORITY="NONE:+VERS-TLS-ALL:+MAC-ALL:+RSA:+AES-128-CBC:+SIGN-ALL:+COMP-NULL" (from https://gnutls.org/manual/html_node/Priority-Strings.html )
to force a pretty basic RSA setup, it still fails the same way.

Any chance you could get an equivalent TLS log from the server?

So in a quick look at the Pulp source, the core architecture is an Apache frontend calling into the Pulp code via WSGI, and the certificate data is passed through and re-parsed and re-validated on the Pulp side via M2Crypto?

Let's first determine where the error is being generated on the server side.  Is the request making it through Apache to the pulp layer at all?

Created attachment 1153585 [details]
http logs syncing an ostree repo from the capsule

Created attachment 1153586 [details]
http logs using curl from a capsule

Colin,

I attached the http logs on a ostree sync from pulp (capsule to server) and a curl command (capsule to server). What other logs can I provide?

It looks as if the request isn't even making it to the apache layer

Created attachment 1153588 [details]
http logs syncing an ostree repo from the capsule

I uploaded the logs syncing the wrong repo. This is the correct satellite logs syncing an ostree repo from capsule to the satellite.

Colin,

What is the status of this bug? Any idea of the timeline to get it fixed? Just wondering as it is blocking https://bugzilla.redhat.com/show_bug.cgi?id=1323644 

Thanks,
John

Can you confirm the relevant error is:

[Tue May 03 17:22:23.179630 2016] [ssl:error] [pid 19033] [client 192.168.121.204:59734] AH02039: Certificate Verification: Error (7): certificate signature failure

If so...a quick read of the OpenSSL code shows there's only one place in the code that returns this error.  Ultimately it's a wrapper for `ASN1_item_verify` - but AFAICS we ignore the error code that the ASN1 layer returns =/  For all we know it could have failed due to OOM.

What seems most likely though is that the client certificate is being somehow corrupted in transit.

This is going to be hard to debug without having a reproducer case to hand myself.  Can I get shell access to that server sufficient to e.g. install a patched openssl?

Just wondering the status of this bug and if it I am correct in understanding it is able to be reproduced?

This one could use a quick glance from the GnuTLS maintainer - Nikos, are you aware of any known issues here?

https://gitlab.com/gnutls/gnutls/issues/39 perhaps?

Any tips/references for tools/strategies to debug this?

That doesn't seem to indicate a problem in gnutls. What is the server running? I could help more if you provide a way to reproduce the issue somehow (e.g., using gnutls-cli).

If the server is running NSS this may be related to:
https://bugzilla.redhat.com/show_bug.cgi?id=1238369

John, sorry about the delay on this, but I think we're in a better place to debug this now.  If you or someone else could generate non-expired certs, that would help.

It'd likely be a bit easier to attach them as one zip file or so.

Stef, does this sound like the issue Cockpit was hitting too?

Colin,

Sounds good, working now to get you new certs and/or access to a machine to debug with. I'll update accordingly.

Created attachment 1175089 [details]
ostree certs and repo config for reproducing

To clarify,
https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=499851
is ~equivalent to 2016.6 for the pull code.

https://wiki.centos.org/SpecialInterestGroup/Atomic/Devel
tracks git master.

For Fedora,
https://bodhi.fedoraproject.org/updates/FEDORA-2016-244f34363c has been outstanding for a while with no karma, but please try that instead of older versions.

(In reply to Nikos Mavrogiannopoulos from comment #31)
> That doesn't seem to indicate a problem in gnutls. What is the server
> running? I could help more if you provide a way to reproduce the issue
> somehow (e.g., using gnutls-cli).

Is the latest zip file here usable for you?  I find it most convenient to use "pet" Docker containers to test out ostree, but you can also just install it via yum on your workstation, or a (remote) VM etc.

> If the server is running NSS this may be related to:
> https://bugzilla.redhat.com/show_bug.cgi?id=1238369

That's a very useful hint.  I do see:

gnutls[2]: sign handshake cert vrfy: picked RSA-SHA512 with SHA512

Is nss advertising support for SHA512 but actually only handling SHA256?

I tried to trick gnutls into doing only sha256 via:

env GNUTLS_DEBUG_LEVEL=9 G_TLS_GNUTLS_PRIORITY="NONE:+VERS-TLS-1.2:+MAC-SHA256:+RSA:+AES-128-CBC:+SIGN-RSA-SHA256:+COMP-NULL" ostree --repo=satellite pull --commit-metadata-only --mirror satellite

but wasn't successful (it stil gave me the same output).

Okay, this should help by dropping ostree out of the picture and just looking at the libsoup -> glib-networking -> gnutls) layers:

https://gist.github.com/cgwalters/065d68bb6122bb500b4abd2f6fd5815e

Now, my testing environment for client certs is an Atomic Host system I have configured to talk to the stage CDN.

On that system I did:
atomic run centos/tools
yum -y install 'pkgconfig(libsoup)'
cd /root
curl -L -o soup-get.c https://gist.githubusercontent.com/cgwalters/065d68bb6122bb500b4abd2f6fd5815e/raw/82e843b37fc633c95fb60a68bfaf83f0badcdb9d/soup-get.c
gcc -Wall $(pkg-config --cflags --libs libsoup-2.4) -o soup-get ./soup-get.c

Now, try to fetch without a client cert:
env GSETTINGS_BACKEND=memory ./soup-get -h --ca-file /host/etc/rhsm/ca/redhat-uep.pem https://cdn.stage.redhat.com/content/dist/rhel/atomic/7/7Server/x86_64/ostree/repo/config
/content/dist/rhel/atomic/7/7Server/x86_64/ostree/repo/config: 6 Server required TLS certificate (no handshake status)

Provide the keys:
env GSETTINGS_BACKEND=memory ./soup-get -h --ca-file /host/etc/rhsm/ca/redhat-uep.pem --cert /host/etc/pki//entitlement/2867616212724431168.pem --key /host/etc/pki/entitlement/2867616212724431168-key.pem  https://cdn.stage.redhat.com/content/dist/rhel/atomic/7/7Server/x86_64/ostree/repo/config
/content/dist/rhel/atomic/7/7Server/x86_64/ostree/repo/config: 200 O K

This client tool also gives the same error when talking to satellite.

(In reply to Colin Walters from comment #45)
> (In reply to Nikos Mavrogiannopoulos from comment #31)
> > That doesn't seem to indicate a problem in gnutls. What is the server
> > running? I could help more if you provide a way to reproduce the issue
> > somehow (e.g., using gnutls-cli).
> 
> Is the latest zip file here usable for you?  I find it most convenient to
> use "pet" Docker containers to test out ostree, but you can also just
> install it via yum on your workstation, or a (remote) VM etc.
> 
> > If the server is running NSS this may be related to:
> > https://bugzilla.redhat.com/show_bug.cgi?id=1238369
> 
> That's a very useful hint.  I do see:
> 
> gnutls[2]: sign handshake cert vrfy: picked RSA-SHA512 with SHA512
> 
> Is nss advertising support for SHA512 but actually only handling SHA256?
> 
> I tried to trick gnutls into doing only sha256 via:
> 
> env GNUTLS_DEBUG_LEVEL=9
> G_TLS_GNUTLS_PRIORITY="NONE:+VERS-TLS-1.2:+MAC-SHA256:+RSA:+AES-128-CBC:
> +SIGN-RSA-SHA256:+COMP-NULL" ostree --repo=satellite pull
> --commit-metadata-only --mirror satellite

I can reproduce it using:
$ gnutls-cli sat-r220-10.lab.eng.rdu2.redhat.com --insecure --x509keyfile ueber.key --x509certfile ueber.cert --priority NORMAL:-SIGN-RSA-SHA512:-SIGN-RSA-SHA384:-SIGN-RSA-SHA224:-SIGN-RSA-SHA256

*** Fatal error: A TLS fatal alert has been received.
*** Received alert [51]: Decrypt error
*** Handshake has failed
GnuTLS error: A TLS fatal alert has been received.

|<4>| HSK[0x559d8a0655f0]: Server's version: 3.3
|<4>| HSK[0x559d8a0655f0]: Selected cipher suite: ECDHE_RSA_AES_256_GCM_SHA384
|<2>| sign handshake cert vrfy: picked RSA-SHA1 with SHA1

So the signature algorithm is not an issue.

Let's try reproducing with local openssl:
$ openssl version
OpenSSL 1.0.2h-fips  3 May 2016

$ openssl s_server -cert cert-rsa.pem -key key-rsa.pem -port 5556
Using default temp DH parameters
ACCEPT


$ gnutls-cli localhost --insecure --x509keyfile ueber.key --x509certfile ueber.cert --priority NORMAL -p 5556 -d 6
|<4>| HSK[0x5653655c0a80]: Server's version: 3.3
|<4>| HSK[0x5653655c0a80]: Selected cipher suite: ECDHE_RSA_AES_256_GCM_SHA384
|<4>| HSK[0x5653655c0a80]: Selected ECC curve SECP256R1 (2)
|<2>| sign handshake cert vrfy: picked RSA-SHA512 with SHA512


and works...

So there is something is going on the server side. Is this apache instance at sat-r220-10.lab.eng.rdu2.redhat.com using mod_ssl or mod_nss? Which version of those?

I tried openssl 1.0.1, and gnutls still interoperates, so knowing what the apache runs is crucial.
/home/nmavrogi/rhel/openssl/openssl-1.0.1e/apps/openssl version
OpenSSL 1.0.1e-fips 11 Feb 2013


A work around for this server is to disable TLS 1.2 and use TLS 1.1 (via priority: "NORMAL:-VERS-TLS1.2")

Nikos, see https://bugzilla.redhat.com/show_bug.cgi?id=1330753#c17

Though someone with more knowledge about the Satellite codebase could answer better.

It seems I figured the issue. The certificate send by gnutls has different size than the one sent by openssl or nss. If I compare the output of the certificates sent using certlint-x509helper, I get something like:
-                <extnID>1.3.6.1.4.1.2312.9.1.24290245979.1</extnID>
+                <extnID>1.3.6.1.4.1.2312.9.1.1467399257435.1</extnID>

It seems that there is an in libtasn1 handling OID elements >2^32. If you'd like a fix on rhel-7 please file a bug as soon so I could fix that issue within the rhel-7.3 deadline.

Let's just move this one.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2551.html