We found another KDF function in libreswan that is not using the NSS KDF API.

Unfortunately, it seems the existing IKE KDF's in NSS are not usable for the Quick Mode use.
 
The libreswan code is in compute_proto_keymat() and the specification is in  https://tools.ietf.org/html/rfc2409#section-5.5
 
 It needs:

   KEYMAT = prf(SKEYID_d, [g(qm)^xy ] | protocol | SPI | Ni_b | Nr_b).
 
which an be thought of as:
 
  KEYMAT = prf(KEY, [KEY] | BYTES)
 
but with the kicker that it also does multiple rounds aka key expansion:
 
      KEYMAT = K1 | K2 | K3 | ...
 
      where
 
        K1 = prf(KEY, [KEY] | BYTES)
        K2 = prf(KEY, K1 | [KEY] | BYTES)
        K3 = prf(KEY, K1 | [KEY] | BYTES)
        etc.
 
to generate the needed keying material >PRF size
 
looking at NSS:

sftk_ike_prf() seems to be the closest; it can compute:
   2. prf(inKey, Ni|Nr); (bDataAsKkey=FALSE, bRekey=FALSE)
   3. prf(inKey, newKey | Ni | Nr); (bDataAsKey=FALSE, bRekey=TRUE) which look good except:
  - it doesn't do multiple rounds aka key expansion

sftk_ike1_prf() can compute:
     prf(inKey, gxyKey || CKYi || CKYr || key_number)

It's problems are:
  - key number byte isn't an option
  - the gxyKey isn't optional
  - it doesn't do multiple rounds aka key expansion (technically the RFC says it should, but I'm not sure it matters)

sftk_ike1_appendix_b_prf() can only compute:
  K1 = prf(K, 0x00)
  K2 = prf(K, K1)
so it cannot be used either (it does do multiple rounds but with wrong values)

sftk_ike_prf_plus() is like sftk_ike1_appendix_b_prf() - multiple
rounds but with wrong values so it too cannot be used.