Bug 689919

release flag should be set to 6.2.0

Copying here what I reported in https://bugzilla.redhat.com/show_bug.cgi?id=689918

Results of an attempt to build nss without softoken or util sources in the tree aren't promissing. Several higher-level libraries and test tools need access to module-private headers from util, freebl, and softoken in order to compile.
 
libraries:
 1. lib/pk11wrap and sysinit use softoken's pkcs11 parser "header"
  NOTE: There are request to move the parser to utils but this will be at best in a future release of nss

 2. lib/pk11warap/dev3hack.c includes pk11init.h
 3. lib/pk11wrap/pk11akey.c uses pk11init.h
 4. lib/pk11wrap/pk11util.c uses pk11init.h
 5. lib/cryptohi/seckey.c uses ec.h
 6. lib/ssl3/ssl3ecc.c uses ec.h
 7. lib/ckfw/pem/anchor.c uses softoken.h
 NOTE: For 5/6/7 We may be able to export ec.h as a public headers

 8. lib/ckfw/pem/ckpem.h uses lowkeyti.h via softoknt.h:
 9. lib/nssysinit.c includes pk11pars.h and so needs pk11init.h
 NOTE: Even if we move pk11pars.h to util we still have the
 problem of accessing pk11init.h which is at a higher level

 10. lib/base/arena.c indirectly includes nssilock.h from util
     and nssilock.h in turn includes utilrename.h and nssilckt.h
 11. lib/nss/utilwrap.c includes templates.c from util
 NOTE: That's a bit problematic, it's code not an exporatble header
 I did a bit of a hack

tools ---
 12. cmd/pk11util/pk11util.c uses pk11init.h
 13. cmd/bltest/blapitest.c needs secrng (build it with nss-ssoftokn?)
 NOTE: 
       blapitest should defintely be built as part of nss-softoken
       and I pan to do that but for a 3.13.x release.
       I will enter a bug upstream to that efect. We now have
       Mozilla bug 172051 approved which enables this as
       blapi test will no longer depend on higher level nss.
       
 14. cmd/fipstest/blapitest.c needs ec.h, lowkeyi.h, lowkeyti.h, softoken.h
 NOTE: Like in item 13 blapitest will be build with nss-softokn
 already doing this in Fedora but not in a clean manner.
 One big problem is the fact that NSS supports at runtime 
 configuring ssl in a bypass mode. Bypass means that ssl/tls 
 can be configured to bypass the pkcs11 layert and call freebl 
 directly. This shoudl also be configured at build time. 
 This a major piece of work to be conducted upstream.

 15. cmd/rsaperf/rsaperf.c lowkeyi.h

We may be able to move ec.h, moving other headers more problematic.
NSS has the concepr of private exports. These are headers we don't
export but are visible from other modules of nss. This no longer works
when we have separete packages.

Below is a hack I deid in the spec file in order to compile as muck as I could and be able to discover all the unwelcome dependencies I listed above.

As part of the %prep section
# Begin Coping hack -----------------------------------------
cp ./mozilla/security/nss/lib/softoken/pk11pars.h ./mozilla/security/nss/lib/pk11wrap/
cp ./mozilla/security/nss/lib/softoken/pk11pars.h ./mozilla/security/nss/lib/sysinit/
cp ./mozilla/security/nss/lib/softoken/pk11init.h ./mozilla/security/nss/lib/pk11wrap/
cp ./mozilla/security/nss/lib/freebl/ec.h ./mozilla/security/nss/lib/pk11wrap/
cp ./mozilla/security/nss/lib/softoken/pkcs11ni.h ./mozilla/security/nss/lib/pk11wrap/
cp ./mozilla/security/nss/lib/softoken/pkcs11ni.h ./mozilla/security/nss/lib/pk11wrap/
cp ./mozilla/security/nss/lib/freebl/ec.h ./mozilla/security/nss/lib/cryptohi/
cp ./mozilla/security/nss/lib/freebl/ec.h ./mozilla/security/nss/lib/ssl/
cp ./mozilla/security/nss/lib/softoken/softoken.h ./mozilla/security/nss/lib/ckfw/pem/
cp ./mozilla/security/nss/lib/softoken/lowkeyti.h ./mozilla/security/nss/lib/ckfw/pem/
cp ./mozilla/security/nss/lib/softoken/softoknt.h ./mozilla/security/nss/lib/ckfw/pem/
cp ./mozilla/security/nss/lib/softoken/pk11init.h ./mozilla/security/nss/lib/sysinit/
cp ./mozilla/security/nss/lib/util/nssilock.h ./mozilla/security/nss/lib/base
cp ./mozilla/security/nss/lib/util/utilrename.h ./mozilla/security/nss/lib/base
cp ./mozilla/security/nss/lib/util/nssilckt.h ./mozilla/security/nss/lib/base

# remove the #include "templates.c" at the end and append it's contents instead
find . -name utilwrap.c
sed -e 's/#include "templates.c"//' ./mozilla/security/nss/lib/nss/utilwrap.c > tmp
mv tmp ./mozilla/security/nss/lib/nss/utilwrap.c
find . -name utilwrap.c
cat ./mozilla/security/nss/lib/util/templates.c >> ./mozilla/security/nss/lib/nss/utilwrap.c
# Hope no one minds that bit of on-the-fly code modification

# -- tools:
cp ./mozilla/security/nss/lib/freebl/secrng.h ./mozilla/security/nss/cmd/bltest/

# remove subdirectories that we don't want
rm -rf ./mozilla/security/nss/lib/util
rm -rf ./mozilla/security/nss/lib/freebl
rm -rf ./mozilla/security/nss/lib/softoken

# remove blapitest because it relies on an unexported function
# blapitest will be part of nss-softokn anyway
rm -rf ./mozilla/security/nss/cmd/bltest
# End Coping hack -----------------------------------------

As stated, this was to create the problem list and it's not something that I would be doing in the spec file. 

As a long term goal it may nice to be bale to build just the the higher levels of nss and not recompile what we have already shipped in nss-softokn and nss-util but given the current state attempting to do by doind violence to the build system is not wise in my opnion and less statisfactory than what we are currently doing. I recommend not "fixing" this for RHEL 6.2.

To clarify on the need info. In order to get nss (the rest of nss) to compile I am forced to copy a bunch of files and I certainly don't want to do this. Is there a somwewhat cleaner way to accomplish it given the current state of the sources?

We really have a couple of options here, and each file is different. For each header ask:

1) is it really needed in the given context? If not remove the include.
2) is the needed symbol in the wrong header file?
    For instance anchor.h and ckpem.h including softoken.h, what symbol do the really need, and should that symbol be included in some already exported file (possibly in util).
3) is the header file itself in the wrong module (pk11pars.h comes to mind).
    I think glenn was a little to agressive in moving files when he should have just moved symbols. secmodt.h, for instance, should never have been moved to softoken.
4) should the header itself actually be exported. ec.h is a likely candidate.

We should feel free to move symbols, and files if we need to, with the caviat that we can't really tweak with softoken that much. What we should do is build a version of what we *would* want if we could change softoken. That should be pushed upstream so it's picked up in RHEL 7. RHEL 5 is already hosed, so it's ok to keep whatever hacks we have there. For RHEL 6, we should feel free to 'copy' a file out of softoken rather than 'move' it. The tricky part comes if we 'copy' a symbol and wind up with conflicts with softoken's copy that we can't change. I suggest we resolve those after we build 'nirvana'.

bob

Created attachment 520484 [details]
spec file changes to build without freebl/softoken/util sources in the tree

Created attachment 520485 [details]
patch sources, config, manifest and makefiles

The changes are for RHEL 6 building only not to be submitted upstream. Maybe some.

The section with Index: ./mozilla/security/nss/lib/softoken/softokn.def
with the  NSS_3.12.10 { # NSS 3.12.10 release" was experimental and forgot to remove it. In blapitest.c I had #idef out the call to sftk_fipsPowerUpSelfTest
because this is not a public call and I doubt we would make it public. On teh other hand when I build blapitest as part of the nss-softoken build there is no problem. I'm doing that in Fedora but with some hacks. Once we update to 3.13 we will do it cleanly. And be able to comp[ile fipstest.c as well.

Created attachment 520486 [details]
change way we build pem module so it  prefers system libraries

Created attachment 520488 [details]
Add missing defines that ssl needs

This is because we are at nss-softokn 3.12.9 and ssl needs them. blapit.h since 3.12.10 has them.

Created attachment 520489 [details]
Skip the magling error test

We can't run this particular error test because there is no softokn3.so in mozilla/dist/Linux..../lib directory for the too to mangle.

Comment on attachment 520484 [details]
spec file changes to build without freebl/softoken/util sources in the tree

With this changes and the patches I was able to do a brew scratch build. I had to disable of a couple of tests as mention elsewhere.
Filed some upstream bugs. May do more as I understand dependencies better.

Comment on attachment 520488 [details]
Add missing defines that ssl needs

r+ rrelyea

Though it may be better to add these to a private header file is ssl.h..

Comment on attachment 520489 [details]
Skip the magling error test

r+, though I'd like to see you getting a copy of the real softoken and try mangling it to make sure integrity is still working.

Comment on attachment 520485 [details]
patch sources, config, manifest and makefiles

r+,

1. though you should probably put the coreconf changes under ifdef so we can push it upstream.

2. We aren't going to export the private symbols in softoken. blapi statically links with softoken.. Perhaps we should move blapi-test to nss-softoken in the future?

Comment on attachment 520484 [details]
spec file changes to build without freebl/softoken/util sources in the tree

r+ for now, but we aren't complete until we straighten out the header file situation.

(In reply to comment #16)
I was able to copy the system token into the dist via the spec file but that caused the fips test to fail I tried it via fips.sh copying into the mangling directory and had the same fips test failures.

Misstyped: s/system token/softoken.so/

Also, in Rawhide I am able to run the cipher suite (blapitest) as part of nss-softoken build itslef. I had to do some modidifications to blapitest and create a set of the libsectool.a that only depends on softoken and below. I use for testing and don't istall it in the system and is a temprary workaround. Yes, it's a bit hackish and can't bring it into rhel 6 for nss-softoken, certainly not at this stage. The changes in nss 3.13 will make things a lot easier.

Since RHEL 6.2 External Beta has begun, and this bug remains
unresolved, it has been rejected as it is not proposed as
exception or blocker.

Red Hat invites you to ask your support representative to
propose this request, if appropriate and relevant, in the
next release of Red Hat Enterprise Linux.

Reflagging for review for RHEL 6.3 inclusion.

Since the revised patches rely on nss 3.13.x, and we haven't yet updated to it, let us continue the review on Bug 689918 for the time being. I have submitted revised and additional patches there for review.

From the review comments on Bug 689918 a lot of work is required to complete this task. Considering also the more pressing tasks that both the reviewer and the submitter have ahead them it is not realistic goal for rhel-6.3. We could change Target Release to rhel-6.4 at the earliest.

(In reply to comment #26)
Replace could with should. Clearing all flags and setting on rhel-6.4?.

Comment on attachment 520486 [details]
change way we build pem module so it  prefers system libraries

r- everything looks correct except the addition of the -lsoftokn in the last file.

bob

Created attachment 888122 [details]
deals with using older version of nss-softokn

This a local patch until update nss-softokn. It's a modified version of the one we applied on rhel-5.

Created attachment 888123 [details]
specfile changes to build without softoken - in patch format

Created attachment 907311 [details]
Additional spec changes to truly build with softoken etc. removed

There were two mistakes which I found by examining intermediate steps.  In the %build section we have
-##### phase 1: remove util/freebl/softoken and low level tools
-######## Remove freebl, softoken and util
-%{__rm} -rf ./mozilla/securitynss/lib/freebl
....
and for the tools

The mozilla/security/ part in the patch is incorrect as we have now a shallower sorce tree. I corrcted this and rechecked and it wan't enough. Did an rhpkg %prep and then ls nss-3.16.1/nss/lib which showed util, freebl, and softoken still there and with ls nss-3.16.1/nss/cmd bltest, bltest, and rsaperf_low were still there. Removal must be done in the %prep phase as the very last set of actions after some copying of files have been done. That's why we have prepping phase after all. These changes accomplish that and the directories are indeed removed as we want.

Comment on attachment 907311 [details]
Additional spec changes to truly build with softoken etc. removed

r+

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1378.html