Can you please specify what is the exact order of checking the REST CAs? 
Also, are there some fallbacks? e.g. if the CA defined in [virt-viewer] is not present, use the system-wide setting?

1) Use [ovirt] 'ca' field
2) Use CA from --ovirt-ca-file command line parameter
3) Use CAs from system-wide store

I don't think we try to use the CA from [virt-viewer] 'ca' for the REST communication. The issue here is that we have an invalid CA in [ovirt] 'ca', I think at some point this was working fine with the same setup.

The thing is that the most common case in oVirt is to have the same CA for REST and host communication. This is the reason why we are providing the CA= in the [ovirt] section. 
If you have a different certificate for the REST for some reason, you need to specify this system wide on client system.
Now, the problem is that in oVirt side we don't know anything about this custom certificate so we can not provide it in the .vv file. 

I think the best solution would be to have the CA from the .vv file as a fallback if you don't find it in system (e.g. trust system first and if not working, than the .vv file).
What you think?

(In reply to Tomas Jelinek from comment #4)
> If you have a different certificate for the REST for some reason, you need
> to specify this system wide on client system.
> Now, the problem is that in oVirt side we don't know anything about this
> custom certificate so we can not provide it in the .vv file. 

Yes, I understand this, and I know that if a custom certificate has been set, then it has to be in the system trust store. However, I would have expected that when a custom certificate has been set, then ovirt/ca is not set to an invalid value.

> I think the best solution would be to have the CA from the .vv file as a
> fallback if you don't find it in system (e.g. trust system first and if not
> working, than the .vv file).
> What you think?

I don't really like making random guesses like these, especially as the behaviour would be very different than the behaviour we have for the SPICE certificate.

(In reply to Christophe Fergeau from comment #5)
> (In reply to Tomas Jelinek from comment #4)
> > If you have a different certificate for the REST for some reason, you need
> > to specify this system wide on client system.
> > Now, the problem is that in oVirt side we don't know anything about this
> > custom certificate so we can not provide it in the .vv file. 
> 
> Yes, I understand this, and I know that if a custom certificate has been
> set, then it has to be in the system trust store. However, I would have
> expected that when a custom certificate has been set, then ovirt/ca is not
> set to an invalid value.

The problem is that when the custom certificate is used, we have no way to find this out on the engine side - it is basically just replacing the certificate in apache.

> 
> > I think the best solution would be to have the CA from the .vv file as a
> > fallback if you don't find it in system (e.g. trust system first and if not
> > working, than the .vv file).
> > What you think?
> 
> I don't really like making random guesses like these, especially as the
> behaviour would be very different than the behaviour we have for the SPICE
> certificate.

We have one more option here - instead of sending the CA directly, we could send an URL where you can find the proper CA. It is always served as http(s)://<addr>/ovirt-engine/ca.crt - but to be more generic, we can send you the url. In that case it will not be a guess of the certificate but a correct location.
What you think about this option?

(In reply to Tomas Jelinek from comment #6)
> (In reply to Christophe Fergeau from comment #5)
> > (In reply to Tomas Jelinek from comment #4)
> > > If you have a different certificate for the REST for some reason, you need
> > > to specify this system wide on client system.
> > > Now, the problem is that in oVirt side we don't know anything about this
> > > custom certificate so we can not provide it in the .vv file. 
> > 
> > Yes, I understand this, and I know that if a custom certificate has been
> > set, then it has to be in the system trust store. However, I would have
> > expected that when a custom certificate has been set, then ovirt/ca is not
> > set to an invalid value.
> 
> The problem is that when the custom certificate is used, we have no way to
> find this out on the engine side - it is basically just replacing the
> certificate in apache.
> 
> > 
> > > I think the best solution would be to have the CA from the .vv file as a
> > > fallback if you don't find it in system (e.g. trust system first and if not
> > > working, than the .vv file).
> > > What you think?
> > 
> > I don't really like making random guesses like these, especially as the
> > behaviour would be very different than the behaviour we have for the SPICE
> > certificate.
> 
> We have one more option here - instead of sending the CA directly, we could
> send an URL where you can find the proper CA. It is always served as
> http(s)://<addr>/ovirt-engine/ca.crt - but to be more generic, we can send
> you the url. In that case it will not be a guess of the certificate but a
> correct location.
> What you think about this option?

ok, so this is not an option - if the certificate is replaced on apache level, the ovirt-engine/ca.crt is serving the engine cert anyway.

What we could do here is to have a new vdc_option parameter where the custom certificate would be stored.

So, if the user decides to replace the certificate on apache level, he will have to also upload this also to vdc_options using engine-config.
@Sandro: would this be an option?

(In reply to Tomas Jelinek from comment #7)

> So, if the user decides to replace the certificate on apache level, he will
> have to also upload this also to vdc_options using engine-config.
> @Sandro: would this be an option?

I'm not a PKI expert and I don't really know how .vv files are generated.
I guess that if .vv are generated with info stored in vdc_options, using engine-config may help.

Googling around I've found:

http://comments.gmane.org/gmane.comp.emulators.ovirt.user/12353

https://bugzilla.redhat.com/show_bug.cgi?id=1125933

which are both related to certificate change but I couldn't find anybody asking about .vv files. Didi, any comment?

The .vv files are generated by us on backend and we need to fill the correct CA to it (in order to let remote-viewer talk to our REST). The question is where to get the CA from if the user replaces it by himself.

An option is to ask the user to put it also to the vdc_options so we can find it there, and the question is if this is acceptable solution or there is some better.

oVirt 4.0 Alpha has been released, moving to oVirt 4.0 Beta target.

Verify in:
engine: 4.0.2.3-0.1.el7ev
host:
OS Version:RHEL - 7.2 - 9.el7
OS Description: Red Hat Enterprise Linux Server 7.2 (Maipo)
Kernel Version:3.10.0 - 327.el7.x86_64
KVM Version:2.3.0 - 31.el7_2.20
LIBVIRT Version:libvirt-1.2.17-13.el7_2.5
VDSM Version:vdsm-4.18.9-1.el7ev
SPICE Version:0.12.4 - 15.el7_2.1

Steps:
1. Install engine and don't provide custom CA and open a console, the .vv file will contain the CA provided by the engine
in both sections: [virt-viewer] and [ovirt]
​2. With custom CA ​and connect to console, the .vv file will contain CA 
only in section: [virt-viewer] and NOT [ovirt]

Results:
Case one: PASS
Case two: PASS


NOTE: DONE with 2 diff engines (one with custom CA)

Case one: vv file
[virt-viewer]
type=spice
host=10.35.161.161
port=-1
password=N05cERXyAwvu
# Password is valid for 120 seconds.
delete-this-file=1
fullscreen=0
title=test-1:%d
toggle-fullscreen=shift+f11
release-cursor=shift+f12
secure-attention=ctrl+alt+end
tls-port=5901
enable-smartcard=0
enable-usb-autoshare=1
usb-filter=-1,-1,-1,-1,0
tls-ciphers=DEFAULT
host-subject=O=scl.lab.tlv.redhat.com,CN=virt-nested-vm14.scl.lab.tlv.redhat.com
ca=-----BEGIN CERTIFICATE-----\nMIIEAzCCAuugAwIBAgICEAAwDQYJKoZIhvcNAQEFBQAwXTELMAkGA1UEBhMCVVMxHzAdBgNVBAoT\nFnNjbC5sYWIudGx2LnJlZGhhdC5jb20xLTArBgNVBAMTJHFlLXhpbzEuc2NsLmxhYi50bHYucmVk\naGF0LmNvbS4xOTYwNTAeFw0xNjA3MjYwNjM3MjRaFw0yNjA3MjUwNjM3MjRaMF0xCzAJBgNVBAYT\nAlVTMR8wHQYDVQQKExZzY2wubGFiLnRsdi5yZWRoYXQuY29tMS0wKwYDVQQDEyRxZS14aW8xLnNj\nbC5sYWIudGx2LnJlZGhhdC5jb20uMTk2MDUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIB\nAQDkQZxoZIS+CEy2/vGeGwCIEEv7wEHh6ZPgbawEXiibksHgDVOt/vGRInw+MSb93Pr9mYGG+JBk\n09O27YYlHaQwInZQbb1cGyvqP86QD1p22zMypq/OluqTWNAeVQ3aesIWYchljDxvJsa1LO/ZKRi6\nOIHe9QO8GGfw9/6+xbjkjoXZZ67PnDGjYOwNxVqR4NK/+Rtvu/YPQmmhbwOSse8KpWjyiGyto21X\n9nJlS06BdzXJ7hk02SeyoydITrxjIAy68apgRuFZP6CGdCXh2AeeZ+aFAduumtHsnhFnHb3FJVl5\n36fBbyjNPT2UHf4GEkHuKrBoD8RoSumuv2vrnNJtAgMBAAGjgcwwgckwHQYDVR0OBBYEFCjCTYoA\nQ4ZCLObZxk3NTKpyAafDMIGGBgNVHSMEfzB9gBQowk2KAEOGQizm2cZNzUyqcgGnw6FhpF8wXTEL\nMAkGA1UEBhMCVVMxHzAdBgNVBAoTFnNjbC5sYWIudGx2LnJlZGhhdC5jb20xLTArBgNVBAMTJHFl\nLXhpbzEuc2NsLmxhYi50bHYucmVkaGF0LmNvbS4xOTYwNYICEAAwDwYDVR0TAQH/BAUwAwEB/zAO\nBgNVHQ8BAf8EBAMCAQYwDQYJKoZIhvcNAQEFBQADggEBAIztvtggBcAevm5psex0Ft7RxSrlFKaR\n+6kLyd2oRz1QaNIMmz+vgHlOofyIhC8Ah+0iJXwlET0xCoEpp5wmRNr6jcw9kMGp9jjelXuhH3V2\n1zammBJcEntLMY/CofNIF/Bz1oGW4MBqYmSZykO7Ff/4e6/SlztXMwFrpaba2r+5NP9LOYKhuPXq\nq08Ljhd/0t509Lgq59sSFPl/kcEh9TjjbzENMeuWnTlJqX/7P7xPn+mzvmqN9eou2CrKfNs5A9Ih\nfme+Wuj6fa52sqQbQRagJfx/zQZFhbH+01ohAs1RW3osVLgz8Wcu9nnizh2HE/p347b3TYFYB6QO\n4LsCWBA=\n-----END CERTIFICATE-----\n
secure-channels=main;inputs;cursor;playback;record;display;smartcard;usbredir
versions=rhev-win64:2.0-128;rhev-win32:2.0-128;rhel7:2.0-6;rhel6:2.0-14
newer-version-url=https://qe-xio1.scl.lab.tlv.redhat.com/ovirt-engine/rhev/client-resources

[ovirt]
host=qe-xio1.scl.lab.tlv.redhat.com
vm-guid=30e7ba76-c93e-48e2-a817-0c09048b1fc7
sso-token=_eq9hnojgTaLatDK67xCNMf_8ygAiCKrY48IW_zxLaV4TL9_ecYiExBI61Q9N17MeD3Opefk6fomsmh-5xLTPQ
admin=1
ca=-----BEGIN CERTIFICATE-----\nMIIEAzCCAuugAwIBAgICEAAwDQYJKoZIhvcNAQEFBQAwXTELMAkGA1UEBhMCVVMxHzAdBgNVBAoT\nFnNjbC5sYWIudGx2LnJlZGhhdC5jb20xLTArBgNVBAMTJHFlLXhpbzEuc2NsLmxhYi50bHYucmVk\naGF0LmNvbS4xOTYwNTAeFw0xNjA3MjYwNjM3MjRaFw0yNjA3MjUwNjM3MjRaMF0xCzAJBgNVBAYT\nAlVTMR8wHQYDVQQKExZzY2wubGFiLnRsdi5yZWRoYXQuY29tMS0wKwYDVQQDEyRxZS14aW8xLnNj\nbC5sYWIudGx2LnJlZGhhdC5jb20uMTk2MDUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIB\nAQDkQZxoZIS+CEy2/vGeGwCIEEv7wEHh6ZPgbawEXiibksHgDVOt/vGRInw+MSb93Pr9mYGG+JBk\n09O27YYlHaQwInZQbb1cGyvqP86QD1p22zMypq/OluqTWNAeVQ3aesIWYchljDxvJsa1LO/ZKRi6\nOIHe9QO8GGfw9/6+xbjkjoXZZ67PnDGjYOwNxVqR4NK/+Rtvu/YPQmmhbwOSse8KpWjyiGyto21X\n9nJlS06BdzXJ7hk02SeyoydITrxjIAy68apgRuFZP6CGdCXh2AeeZ+aFAduumtHsnhFnHb3FJVl5\n36fBbyjNPT2UHf4GEkHuKrBoD8RoSumuv2vrnNJtAgMBAAGjgcwwgckwHQYDVR0OBBYEFCjCTYoA\nQ4ZCLObZxk3NTKpyAafDMIGGBgNVHSMEfzB9gBQowk2KAEOGQizm2cZNzUyqcgGnw6FhpF8wXTEL\nMAkGA1UEBhMCVVMxHzAdBgNVBAoTFnNjbC5sYWIudGx2LnJlZGhhdC5jb20xLTArBgNVBAMTJHFl\nLXhpbzEuc2NsLmxhYi50bHYucmVkaGF0LmNvbS4xOTYwNYICEAAwDwYDVR0TAQH/BAUwAwEB/zAO\nBgNVHQ8BAf8EBAMCAQYwDQYJKoZIhvcNAQEFBQADggEBAIztvtggBcAevm5psex0Ft7RxSrlFKaR\n+6kLyd2oRz1QaNIMmz+vgHlOofyIhC8Ah+0iJXwlET0xCoEpp5wmRNr6jcw9kMGp9jjelXuhH3V2\n1zammBJcEntLMY/CofNIF/Bz1oGW4MBqYmSZykO7Ff/4e6/SlztXMwFrpaba2r+5NP9LOYKhuPXq\nq08Ljhd/0t509Lgq59sSFPl/kcEh9TjjbzENMeuWnTlJqX/7P7xPn+mzvmqN9eou2CrKfNs5A9Ih\nfme+Wuj6fa52sqQbQRagJfx/zQZFhbH+01ohAs1RW3osVLgz8Wcu9nnizh2HE/p347b3TYFYB6QO\n4LsCWBA=\n-----END CERTIFICATE-----\n 


Case two vv file:
[virt-viewer]
type=spice
host=10.34.63.223
port=-1
password=oFgXFPUL69kZ
# Password is valid for 120 seconds.
delete-this-file=1
fullscreen=0
title=israel:%d
toggle-fullscreen=shift+f11
release-cursor=shift+f12
secure-attention=ctrl+alt+end
tls-port=5901
enable-smartcard=0
enable-usb-autoshare=1
usb-filter=-1,-1,-1,-1,0
tls-ciphers=DEFAULT
host-subject=O=rhev.lab.eng.brq.redhat.com,CN=10.34.63.223
ca=-----BEGIN CERTIFICATE-----\nMIIELzCCAxegAwIBAgICEAAwDQYJKoZIhvcNAQEFBQAwazELMAkGA1UEBhMCVVMxJDAiBgNVBAoT\nG3JoZXYubGFiLmVuZy5icnEucmVkaGF0LmNvbTE2MDQGA1UEAxMtMTAtMzQtNjAtMTIucmhldi5s\nYWIuZW5nLmJycS5yZWRoYXQuY29tLjQ4Mjk0MB4XDTE2MDUyNDExNDYyNVoXDTI2MDUyMzExNDYy\nNVowazELMAkGA1UEBhMCVVMxJDAiBgNVBAoTG3JoZXYubGFiLmVuZy5icnEucmVkaGF0LmNvbTE2\nMDQGA1UEAxMtMTAtMzQtNjAtMTIucmhldi5sYWIuZW5nLmJycS5yZWRoYXQuY29tLjQ4Mjk0MIIB\nIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAv7smvcgr8oEWcGuR4gPnNGXYKzJBYNvqC22X\ng5HyoHSlsJ4BbhTbEu38C8I+QwKVSlNYcfgHh8q+0/fwPPYuuL1BEtydXehusLuVhQqhMRgTkI55\n1AoJAAN5Mwq1Qz3/lwJI+3o3eLC4YsPOyOSW7EdJsFQq1CLoEblfwA79xmchBmekxDckVpjWgyFI\n8QQ/BTA1eoKNLbBUu21nMUstARtf+g9ubqP+cafC9kpgj3Kggxmj/pdJ00aB1bwJXCoTXue+kZGI\nxY791JSe52XpjZalyGxY/x6IJKf1YzTRhHQncRzD54eI2vBmLU4QRaCSZyGfFpmUkIZayUMs1sma\nqwIDAQABo4HcMIHZMB0GA1UdDgQWBBQ15LnKiWeTUqivsCK9DWwYJFd4dTCBlgYDVR0jBIGOMIGL\ngBQ15LnKiWeTUqivsCK9DWwYJFd4daFvpG0wazELMAkGA1UEBhMCVVMxJDAiBgNVBAoTG3JoZXYu\nbGFiLmVuZy5icnEucmVkaGF0LmNvbTE2MDQGA1UEAxMtMTAtMzQtNjAtMTIucmhldi5sYWIuZW5n\nLmJycS5yZWRoYXQuY29tLjQ4Mjk0ggIQADAPBgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQEAwIB\nBjANBgkqhkiG9w0BAQUFAAOCAQEARdstlv2f8A8PpMDkzFhyyIuJGrsa/T9G6SjhUmHgpz3+BE36\nWe0adWQO1cNirv5U2rnA2c4DHUxsGE4dbr1IVdRBm9dsOJb/Gt1BVUItaHej6qFj8UrClJHGKoxQ\n8epYLRG3OUpG5iVksWmq08K2ILZaIeXv/pIaGFWQ6wddvqJUBBrwc7sh2ee1XxNV+TSq74KnUQTQ\nOeH/PMFbSgoqt9srFoB1/5kD7N1JsIHZupIjUVpc0qezyHsHQFIvIQVak7rLXw43n9YaYLp8uEKA\nXUNlr/6TXZckiUbBBZ9fyvmYvYbLcjuuBviXhPHOqQppSNvfYsPFp5YPUFoU2RfBAw==\n-----END CERTIFICATE-----\n
secure-channels=main;inputs;cursor;playback;record;display;smartcard;usbredir
versions=rhev-win64:2.0-160;rhev-win32:2.0-160;rhel7:2.0-6;rhel6:99.0-1
newer-version-url=https://jb-rhevm40.rhev.lab.eng.brq.redhat.com/ovirt-engine/rhev/client-resources

[ovirt]
host=jb-rhevm40.rhev.lab.eng.brq.redhat.com
vm-guid=fc959b6c-9d5b-493a-8b17-a2cdaaf36f94
sso-token=HDP9upFc2p9167OKp1qJRyQ8XugjpInN0OUu7M8ha4XcSCaXrzyRv2h7rPGcgr0Z7x8FZRGOpJndhh1qiKU75w
admin=1

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-1743.html