Bug 1492107

Description of problem:
- VNC cannot be used when FIPS is enabled because DH_BITS is too low

Version-Release number of selected component (if applicable):
- 1.8.0



How reproducible:
100%

Steps to Reproduce:
1. set fips=1 on the kernel cmdline of the system hosting the VNC server 
2. boot with this setting and attempt to use vncviewer to connect to a system running a vnc daemon with FIPS turned on
3. Observe error:

$ vncviewer 192.168.1.2:5901

TigerVNC Viewer 64-bit v1.8.0
Built on: 2017-05-17 05:47
Copyright (C) 199-2017 TigerVNC Team and many others (see README.txt)
see http://www.tigervnc.org for information on TigerVNC.

Wed Sep 13 09:39:52 2017
DecodeManager: Detected 4 CPU cores(s)
DecodeManager: Creating 4 decoder thread(s)
CConn:                  connected to host 192.168.1.2 port 5901
CConnection:     Server supports RFB protocol version 3.8
CConnection:     Using RFB protocol version 3.8
CConnection:      Choosing security type VeNCrypt(19)
CVeNCrypt:         Choosing security type TLSVnc (258)
CCon:                   gnutls_dh_params_generate2 failed

Log from VNC Server side:

Wed Sep 13 10:54:38 2017
Connections: accepted: 192.168.1.1::12345
SConnection: Client needs protocol version 3.8
SConnection: Client requests security type VeNCrypt(19)
SVeNCrypt:   Client requests security type TLSVnc (258)
Sconnection: AuthFailureException: gnutls_dh_params_generate2 failed
Connections: closed: 192.168.1.1::12345 (gnutls_dh_params_generate2 failed)
EncodeManager: Framebuffer updates: 0
EncodeManager:     Total: 0 rects, 0 pixels
EncodeManager:                0 B (1: -nan ratio)
TLS:                           TLS session wasn't terminated gracefully
ComparingUpdateTracker: 0 pixels in / 0 pixels out
ComparingUpdateTracker: (1: -nan ratio)



Actual results:
"gnutls_dh_params_generate2" failed then after too many tries get "Too Many Security Errors"

Expected results:
- successful connection

Additional info:

It appears to be the same problem as identified in libvirt here: https://www.redhat.com/archives/libvir-list/2014-September/msg00194.html

This should fix it:

diff -up tigervnc-1.8.0/common/rfb/SSecurityTLS.cxx.moar-bits tigervnc-1.8.0/common/rfb/SSecurityTLS.cxx
--- tigervnc-1.8.0/common/rfb/SSecurityTLS.cxx.moar-bits	2017-09-14 16:32:59.224766365 -0400
+++ tigervnc-1.8.0/common/rfb/SSecurityTLS.cxx	2017-09-14 16:33:20.974546604 -0400
@@ -36,7 +36,7 @@
 #include <rdr/TLSInStream.h>
 #include <rdr/TLSOutStream.h>
 
-#define DH_BITS 1024 /* XXX This should be configurable! */
+#define DH_BITS 2048 /* XXX This should be configurable! */
 
 using namespace rfb;