Bug 2106940

Description of problem:

  After making RHEL 9 FIPS compliant, printing breaks.
  The pdftopdf and bannertopdf filters use QPDFWriter, which
  results in a call to 

    gnutls_hash_init(&this->hash_ctx, GNUTLS_DIG_MD5)

  which fails because FIPS doesn't allow MD5.

  Not sure if this is something that needs to be addressed
  in pdftopdf/bannertopdf (cups-filters) or libqpdf (qpdf).
  I thought I'd file the bug against cups-filters and it
  can be moved if needed.
  
Version-Release number of selected component (if applicable):

  cups-filters-1.28.7-10.el9.x86_64
  qpdf-libs-10.3.1-4.el9.x86_64

How reproducible:

  100%

Steps to Reproduce:

  1. fips-mode-setup --enable
  2. /usr/lib/cups/filter/pdftopdf 1 user title 1 "" /usr/share/cups/data/default-testpage.pdf > /dev/null
  3. /usr/lib/cups/filter/bannertopdf 1 user title 1 "" /usr/share/cups/data/testprint > /dev/null

Actual results:

  # /usr/lib/cups/filter/pdftopdf 1 user title 1 "" /usr/share/cups/data/default-testpage.pdf > /dev/null
  DEBUG: pdftopdf: No PPD file specified, could not determine whether to log pages or not, so turned off page logging. 
  DEBUG: PDF interactive form and annotation flattening done via QPDF
  ERROR: Exception: gnutls: MD5 error: An algorithm that is not enabled was negotiated.

  #  /usr/lib/cups/filter/bannertopdf 1 user title 1 "" /usr/share/cups/data/testprint > /dev/null
  DEBUG: Could not open PPD file '(null)'
  DEBUG: PDF template file doesn't have form. It's okay.
  terminate called after throwing an instance of 'std::runtime_error'
    what():  gnutls: MD5 error: An algorithm that is not enabled was negotiated.
  Aborted (core dumped)

Expected results:

  No errors.
  Printing with FIPS enabled should work

Additional info:

  Some backtraces:

  [root@sf03264105 ~]# gdb /usr/lib/cups/filter/bannertopdf 
  (gdb) set args 1 user title 1 '' /usr/share/cups/data/testprint
  (gdb) run
  Starting program: /usr/lib/cups/filter/bannertopdf 1 user title 1 '' /usr/share/cups/data/testprint

  Program received signal SIGABRT, Aborted.
  0x00007ffff79fb42c in __pthread_kill_implementation () from /lib64/libc.so.6
  (gdb) bt
  #0  0x00007ffff79fb42c in __pthread_kill_implementation () from /lib64/libc.so.6
  #1  0x00007ffff79aed06 in raise () from /lib64/libc.so.6
  #2  0x00007ffff79817d3 in abort () from /lib64/libc.so.6
  #3  0x00007ffff7c1ba01 in __gnu_cxx::__verbose_terminate_handler() [clone .cold] () from /lib64/libstdc++.so.6
  #4  0x00007ffff7c2735c in __cxxabiv1::__terminate(void (*)()) () from /lib64/libstdc++.so.6
  #5  0x00007ffff7c273c7 in std::terminate() () from /lib64/libstdc++.so.6
  #6  0x00007ffff7c27669 in __cxa_throw () from /lib64/libstdc++.so.6
  #7  0x00007ffff7debcf4 in QPDFCrypto_gnutls::MD5_init() [clone .cold] () from /lib64/libqpdf.so.28
  #8  0x00007ffff7df42f1 in MD5::MD5() () from /lib64/libqpdf.so.28
  #9  0x00007ffff7e80529 in QPDF::compute_data_key(std::__cxx11::basic_string<char, std::char_traits<char>,
      std::allocator<char> > const&, int, int, bool, int, int) () from /lib64/libqpdf.so.28
  #10 0x00007ffff7e6649d in QPDFWriter::setDataKey(int) () from /lib64/libqpdf.so.28
  #11 0x00007ffff7e6c446 in QPDFWriter::writeObject(QPDFObjectHandle, int) () from /lib64/libqpdf.so.28
  #12 0x00007ffff7e77dba in QPDFWriter::write() () from /lib64/libqpdf.so.28
  #13 0x000055555555cc83 in pdf_write (file=<optimized out>, pdf=0x55555556c640) at filter/pdf.cxx:352
  #14 generate_banner_pdf (banner=<optimized out>, ppd=<optimized out>, jobid=0x7fffffffe1c0 "\001", 
      user=<optimized out>, jobtitle=0x7fffffffe160 "\001SVUUU", noptions=<optimized out>, options=0x7fffffffe3d0)
      at filter/bannertopdf.c:516
  #15 0x000055555555858b in main (argc=<optimized out>, argv=0x7fffffffe538) at filter/bannertopdf.c:563

  [root@sf03264105 ~]# gdb /usr/lib/cups/filter/pdftopdf
  (gdb) set args 1 user title 1 "" /usr/share/cups/data/default-testpage.pdf
  (gdb) break QPDFCrypto_gnutls::MD5_init
  Breakpoint 1 at 0x7ffff7e8ac6a (2 locations)

  (gdb) run
  
  Breakpoint 1, 0x00007ffff7f588c0 in QPDFCrypto_gnutls::MD5_init() () from /lib64/libqpdf.so.28
  (gdb) bt
  #0  0x00007ffff7f588c0 in QPDFCrypto_gnutls::MD5_init() () from /lib64/libqpdf.so.28
  #1  0x00007ffff7e932f1 in MD5::MD5() () from /lib64/libqpdf.so.28
  #2  0x00007ffff7f1f529 in QPDF::compute_data_key(std::__cxx11::basic_string<char, std::char_traits<char>,
      std::allocator<char> > const&, int, int, bool, int, int) () from /lib64/libqpdf.so.28
  #3  0x00007ffff7f0549d in QPDFWriter::setDataKey(int) () from /lib64/libqpdf.so.28
  #4  0x00007ffff7f0b446 in QPDFWriter::writeObject(QPDFObjectHandle, int) () from /lib64/libqpdf.so.28
  #5  0x00007ffff7f16dba in QPDFWriter::write() () from /lib64/libqpdf.so.28
  #6  0x00005555555691e1 in QPDF_PDFTOPDF_Processor::emitFilename (this=<optimized out>, name=<optimized out>)
      at filter/pdftopdf/qpdf_pdftopdf_processor.cc:743
  #7  0x000055555555e23e in main (argc=<optimized out>, argv=<optimized out>) at filter/pdftopdf/pdftopdf.cc:1310

  I think the code that generates the actual error is here in libqpdf/QPDFCrypto_gnutls.cc:

  44 void
  45 QPDFCrypto_gnutls::MD5_init()
  46 {
  47     MD5_finalize();
  48     int code = gnutls_hash_init(&this->hash_ctx, GNUTLS_DIG_MD5);
  49     if (code < 0)
  50     {
  51         this->hash_ctx = nullptr;
  52         throw std::runtime_error(
  53             std::string("gnutls: MD5 error: ") +
  54             std::string(gnutls_strerror(code)));
  55     }
  56 }