Bug 689918 - Build NSS without any softoken or util sources present in the tree
Build NSS without any softoken or util sources present in the tree
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: nss (Show other bugs)
19
All Linux
high Severity high
: ---
: ---
Assigned To: Elio Maldonado Batiz
Fedora Extras Quality Assurance
: Patch
Depends On: 806588 883114
Blocks: 689919
  Show dependency treegraph
 
Reported: 2011-03-22 15:25 EDT by Elio Maldonado Batiz
Modified: 2013-08-03 19:59 EDT (History)
5 users (show)

See Also:
Fixed In Version: nss-util-3.15.1-1.fc18
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 689919 (view as bug list)
Environment:
Last Closed: 2013-06-29 14:17:13 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Main patch for building nss without softoken or util in the tree (15.15 KB, patch)
2012-01-12 10:22 EST, Elio Maldonado Batiz
no flags Details | Diff
Adds a target to the Makefile to copy prebuilt softoken libs to the tree (671 bytes, patch)
2012-01-12 10:24 EST, Elio Maldonado Batiz
no flags Details | Diff
Copy the softoken libs from the buildroot to the build tree. (446 bytes, patch)
2012-01-12 10:26 EST, Elio Maldonado Batiz
no flags Details | Diff
If blapit.h lacks them this header provides them (542 bytes, patch)
2012-01-12 10:39 EST, Elio Maldonado Batiz
no flags Details | Diff
Changes to the spec file (12.30 KB, patch)
2012-01-12 10:46 EST, Elio Maldonado Batiz
no flags Details | Diff
Disable two stress that are currently failing (1.42 KB, patch)
2012-01-12 10:54 EST, Elio Maldonado Batiz
no flags Details | Diff
Disable ssl pkcs #11 bypass at build time (824 bytes, patch)
2012-02-13 17:55 EST, Elio Maldonado Batiz
rrelyea: review-
Details | Diff
Main patch to build nss without softoken or util sources in the tree (14.14 KB, patch)
2012-02-13 17:58 EST, Elio Maldonado Batiz
rrelyea: review+
Details | Diff
Adds a Makefile target to copy prebuild softoken list to build tree (671 bytes, patch)
2012-02-13 18:01 EST, Elio Maldonado Batiz
rrelyea: review+
Details | Diff
Copies the nss-softoken shared lins to the build tree (446 bytes, patch)
2012-02-13 18:03 EST, Elio Maldonado Batiz
rrelyea: review+
Details | Diff
Adds the missing defines blapit.h lacks on rhel-6.x with older softokn/freebl (542 bytes, patch)
2012-02-13 18:05 EST, Elio Maldonado Batiz
rrelyea: review-
Details | Diff
Disable some ssl stress that are failing (1.42 KB, patch)
2012-02-13 18:14 EST, Elio Maldonado Batiz
rrelyea: review-
Details | Diff
Changes to the spec file in patch form (12.67 KB, patch)
2012-02-13 18:17 EST, Elio Maldonado Batiz
rrelyea: review+
Details | Diff
Disable ssl bypass at build time (53 bytes, text/plain)
2012-03-25 15:19 EDT, Elio Maldonado Batiz
no flags Details
Disable ssl pkcs11 layer bypass at build time (13.16 KB, patch)
2012-03-30 18:40 EDT, Elio Maldonado Batiz
rrelyea: review+
Details | Diff
Disable ssl pkcs11 bypass at build time and preserve ABI (18.24 KB, patch)
2012-05-14 14:13 EDT, Elio Maldonado Batiz
no flags Details | Diff
Remove include.h from seckey.c is not needed (446 bytes, patch)
2012-11-11 22:50 EST, Elio Maldonado Batiz
rrelyea: review+
Details | Diff
copies system softoken libraries into to {DIST}/lib in the source tree (446 bytes, application/x-shellscript)
2012-11-11 22:58 EST, Elio Maldonado Batiz
rrelyea: review+
Details
patches build system files to build nss with softoken or util in the tree (11.79 KB, patch)
2012-11-11 23:23 EST, Elio Maldonado Batiz
no flags Details | Diff
Changes to the spec file in patch form (9.09 KB, patch)
2012-11-11 23:30 EST, Elio Maldonado Batiz
no flags Details | Diff
Changes to the spec file in patch form (8.27 KB, patch)
2012-11-12 13:32 EST, Elio Maldonado Batiz
no flags Details | Diff
patch build system files to build without softoken or util in the tree (11.39 KB, patch)
2012-11-12 17:30 EST, Elio Maldonado Batiz
rrelyea: review-
Details | Diff
Changes to the spec file in patch form (8.29 KB, patch)
2012-11-12 17:33 EST, Elio Maldonado Batiz
rrelyea: review+
Details | Diff
patch build system files to build without softoken or util in the tree (7.20 KB, patch)
2012-12-27 17:42 EST, Elio Maldonado Batiz
no flags Details | Diff
split rsaperf into high and low level components (49.23 KB, patch)
2012-12-27 17:48 EST, Elio Maldonado Batiz
no flags Details | Diff
adds /usr/includes/templates to the includes list (642 bytes, patch)
2012-12-27 17:56 EST, Elio Maldonado Batiz
no flags Details | Diff
Changes to the spec file in patch form (7.33 KB, patch)
2012-12-27 17:59 EST, Elio Maldonado Batiz
no flags Details | Diff
copy system softoken libraries and tools to dist in the tree (687 bytes, application/x-shellscript)
2012-12-27 18:04 EST, Elio Maldonado Batiz
no flags Details
changes in patches and spec file to build nss without softoken (10.02 KB, patch)
2013-06-18 17:22 EDT, Elio Maldonado Batiz
no flags Details | Diff
changes to spec file to build nss without softoken (8.23 KB, patch)
2013-06-18 17:37 EDT, Elio Maldonado Batiz
rrelyea: review+
Details | Diff
Pick up templates.c from the nss system templates directory (555 bytes, patch)
2013-06-18 17:40 EDT, Elio Maldonado Batiz
rrelyea: review+
Details | Diff


External Trackers
Tracker ID Priority Status Summary Last Updated
Mozilla Foundation 835919 None None None Never

  None (edit)
Description Elio Maldonado Batiz 2011-03-22 15:25:47 EDT
Description of problem: 
The split of softoken and utils from nss as their own package done on nss-3.12.4 for fedora 12 is incomplete. We still need to carry around the entire sources. See https://bugzilla.redhat.com/show_bug.cgi?id=508479#c16

We compile everything and at when making the rpm then remove the pieces that have already been shipped by nss-softokn and nss-util. nss should be able to build and meet its build dependencies on lower layers from the softokn/util libraries and headers already installed in the build system. This is not currently possible due to some unwanted dependencies of higher level code in nss on private headers of softokn and freebl.

Version-Release number of selected component (if applicable): nss-3.12.9

How reproducible: Always

Steps to Reproduce:
1. Create an nss source tar ball with the softokn/freebl/util sources removed
2. Try to compile nss.
An experimental script can that does such removal is available.
  
Actual results:
It deosn't compile due to sevral missing headers

Expected results:
It builds.

Additional info:
Comment 1 Elio Maldonado Batiz 2011-08-12 19:29:38 EDT
Results of an attempt to build nss without softoken or util sources in the tree are not promissing. As I experemented I added comments to nss.spec. Here I am copying and removinf the #'s at teh beginning.

Higher-level libraries and test tools need access to
module-private headers from util, freebl, and softoken
 
libraries:
 1. lib/pk11wrap and sysinit use softoken's pkcs11 parser "header"
     There are plans to fix this by moving the parser to utils
     but this will be in a future release of nss
 --- we may be able to make ec.h but the other headers
     are more problematic
 2. lib/pk11warap/dev3hack.c includes pk11init.h
 3. lib/pk11wrap/pk11akey.c uses pk11init.h
 4. lib/pk11wrap/pk11util.c uses pk11init.h
 5. lib/cryptohi/seckey.c uses ec.h
 6. lib/ssl3/ssl3ecc.c uses ec.h
  7. lib/ckfw/pem/anchor.c uses softoken.h
 NOTE: For 5/6/7 We may be able to export ec.h as a public headers

  8. lib/ckfw/pem/ckpem.h uses lowkeyti.h via softoknt.h:
  9. lib/nssysinit.c includes pk11pars.h and so needs pk11init.h
 NOTE: Even if we move pk11pars.h to util we still have the
 problem of accessing pk11init.h which is at a higher level

 10. lib/base/arena.c indirectly includes nssilock.h from util
     and nssilock.h in turn includes utilrename.h and nssilckt.h
 11. lib/nss/utilwrap.c includes templates.c from util
 NOTE: That's a bit problematic, it's code not an exporatble header
 I did a bit of a hack

tools ---
 12. cmd/pk11util/pk11util.c uses pk11init.h
 13. cmd/bltest/blapitest.c needs secrng (build it with nss-ssoftokn?)
 NOTE: 
       blapitest should defintely be built as part of nss-softoken
       and I pan to do that but for a 3.13.x release.
       I will enter a bug upstream to that efect. We now have
       Mozilla bug 172051 approved which enables this as
       blapi test will no longer depend on higher level nss.
       
 14. cmd/fipstest/blapitest.c needs ec.h, lowkeyi.h, lowkeyti.h, softoken.h
 NOTE: Like in item 13 blapitest will be build with nss-softokn
 already doing this in Fedora but not in a clean manner.
 One big problem is the fact that NSS supports at runtime 
 configuring ssl in a bypass mode. Bypass means that ssl/tls 
 can be configured to bypass the pkcs11 layert and call freebl 
 directly. This shoudl also be configured at build time. 
 This a major piece of work to be conducted upstream.

 15. cmd/rsaperf/rsaperf.c lowkeyi.h

We may be able to move ec.h, moving other headers more problematic.
NSS has the concepr of private exports. These are headers we don't
export but are visible from other modules of nss. This no longer works
when we have separete packages.

Below is a hack I deid in the spec file in order to compile as muck as I could and be able to discover all the unwelcome dependencies I listed above.

As part of the %prep section
# Begin Coping hack -----------------------------------------
cp ./mozilla/security/nss/lib/softoken/pk11pars.h ./mozilla/security/nss/lib/pk11wrap/
cp ./mozilla/security/nss/lib/softoken/pk11pars.h ./mozilla/security/nss/lib/sysinit/
cp ./mozilla/security/nss/lib/softoken/pk11init.h ./mozilla/security/nss/lib/pk11wrap/
cp ./mozilla/security/nss/lib/freebl/ec.h ./mozilla/security/nss/lib/pk11wrap/
cp ./mozilla/security/nss/lib/softoken/pkcs11ni.h ./mozilla/security/nss/lib/pk11wrap/
cp ./mozilla/security/nss/lib/softoken/pkcs11ni.h ./mozilla/security/nss/lib/pk11wrap/
cp ./mozilla/security/nss/lib/freebl/ec.h ./mozilla/security/nss/lib/cryptohi/
cp ./mozilla/security/nss/lib/freebl/ec.h ./mozilla/security/nss/lib/ssl/
cp ./mozilla/security/nss/lib/softoken/softoken.h ./mozilla/security/nss/lib/ckfw/pem/
cp ./mozilla/security/nss/lib/softoken/lowkeyti.h ./mozilla/security/nss/lib/ckfw/pem/
cp ./mozilla/security/nss/lib/softoken/softoknt.h ./mozilla/security/nss/lib/ckfw/pem/
cp ./mozilla/security/nss/lib/softoken/pk11init.h ./mozilla/security/nss/lib/sysinit/
cp ./mozilla/security/nss/lib/util/nssilock.h ./mozilla/security/nss/lib/base
cp ./mozilla/security/nss/lib/util/utilrename.h ./mozilla/security/nss/lib/base
cp ./mozilla/security/nss/lib/util/nssilckt.h ./mozilla/security/nss/lib/base

# remove the #include "templates.c" at the end and append it's contents instead
find . -name utilwrap.c
sed -e 's/#include "templates.c"//' ./mozilla/security/nss/lib/nss/utilwrap.c > tmp
mv tmp ./mozilla/security/nss/lib/nss/utilwrap.c
find . -name utilwrap.c
cat ./mozilla/security/nss/lib/util/templates.c >> ./mozilla/security/nss/lib/nss/utilwrap.c
# Hope no one minds that bit of on-the-fly code modification

# -- tools:
cp ./mozilla/security/nss/lib/freebl/secrng.h ./mozilla/security/nss/cmd/bltest/

# remove subdirectories that we don't want
rm -rf ./mozilla/security/nss/lib/util
rm -rf ./mozilla/security/nss/lib/freebl
rm -rf ./mozilla/security/nss/lib/softoken

# remove blapitest because it relies on an unexported function
# blapitest will be part of nss-softokn anyway
rm -rf ./mozilla/security/nss/cmd/bltest
# End Coping hack -----------------------------------------
As stated this was in order to discover the problems and create the list. 
It's not something that I recommend doing in the spec file.
Comment 2 Elio Maldonado Batiz 2012-01-11 14:04:55 EST
Pasting here the thread from Bug 689919 whre we made some progress.

Elio Maldonado Batiz 2011-08-24 14:56:01 EDT

To clarify on the need info. In order to get nss (the rest of nss) to compile I
am forced to copy a bunch of files and I certainly don't want to do this. Is
there a somwewhat cleaner way to accomplish it given the current state of the
sources?

[reply] [-]
Private
Comment 8 Bob Relyea 2011-08-24 15:11:11 EDT

We really have a couple of options here, and each file is different. For each
header ask:

1) is it really needed in the given context? If not remove the include.
2) is the needed symbol in the wrong header file?
    For instance anchor.h and ckpem.h including softoken.h, what symbol do the
really need, and should that symbol be included in some already exported file
(possibly in util).
3) is the header file itself in the wrong module (pk11pars.h comes to mind).
    I think glenn was a little to agressive in moving files when he should have
just moved symbols. secmodt.h, for instance, should never have been moved to
softoken.
4) should the header itself actually be exported. ec.h is a likely candidate.

We should feel free to move symbols, and files if we need to, with the caviat
that we can't really tweak with softoken that much. What we should do is build
a version of what we *would* want if we could change softoken. That should be
pushed upstream so it's picked up in RHEL 7. RHEL 5 is already hosed, so it's
ok to keep whatever hacks we have there. For RHEL 6, we should feel free to
'copy' a file out of softoken rather than 'move' it. The tricky part comes if
we 'copy' a symbol and wind up with conflicts with softoken's copy that we
can't change. I suggest we resolve those after we build 'nirvana'.

bob

[reply] [-]
Private
Comment 9 Elio Maldonado Batiz 2011-08-29 19:49:44 EDT

Created attachment 520484 [details]
spec file changes to build without freebl/softoken/util sources in the tree

[reply] [-]
Private
Comment 10 Elio Maldonado Batiz 2011-08-29 20:00:10 EDT

Created attachment 520485 [details]
patch sources, config, manifest and makefiles

The changes are for RHEL 6 building only not to be submitted upstream. Maybe
some.

The section with Index: ./mozilla/security/nss/lib/softoken/softokn.def
with the  NSS_3.12.10 { # NSS 3.12.10 release" was experimental and forgot to
remove it. In blapitest.c I had #idef out the call to sftk_fipsPowerUpSelfTest
because this is not a public call and I doubt we would make it public. On teh
other hand when I build blapitest as part of the nss-softoken build there is no
problem. I'm doing that in Fedora but with some hacks. Once we update to 3.13
we will do it cleanly. And be able to comp[ile fipstest.c as well.

[reply] [-]
Private
Comment 11 Elio Maldonado Batiz 2011-08-29 20:03:52 EDT

Created attachment 520486 [details]
change way we build pem module so it  prefers system libraries

[reply] [-]
Private
Comment 12 Elio Maldonado Batiz 2011-08-29 20:06:56 EDT

Created attachment 520488 [details]
Add missing defines that ssl needs

This is because we are at nss-softokn 3.12.9 and ssl needs them. blapit.h since
3.12.10 has them.

[reply] [-]
Private
Comment 13 Elio Maldonado Batiz 2011-08-29 20:10:44 EDT

Created attachment 520489 [details]
Skip the magling error test

We can't run this particular error test because there is no softokn3.so in
mozilla/dist/Linux..../lib directory for the too to mangle.

[reply] [-]
Private
Comment 14 Elio Maldonado Batiz 2011-08-29 20:16:11 EDT

Comment on attachment 520484 [details]
spec file changes to build without freebl/softoken/util sources in the tree

With this changes and the patches I was able to do a brew scratch build. I had
to disable of a couple of tests as mention elsewhere.
Filed some upstream bugs. May do more as I understand dependencies better.

[reply] [-]
Private
Comment 15 Bob Relyea 2011-08-29 20:34:58 EDT

Comment on attachment 520488 [details]
Add missing defines that ssl needs

r+ rrelyea

Though it may be better to add these to a private header file is ssl.h..

[reply] [-]
Private
Comment 16 Bob Relyea 2011-08-29 20:36:07 EDT

Comment on attachment 520489 [details]
Skip the magling error test

r+, though I'd like to see you getting a copy of the real softoken and try
mangling it to make sure integrity is still working.

[reply] [-]
Private
Comment 17 Bob Relyea 2011-08-29 20:41:01 EDT

Comment on attachment 520485 [details]
patch sources, config, manifest and makefiles

r+,

1. though you should probably put the coreconf changes under ifdef so we can
push it upstream.

2. We aren't going to export the private symbols in softoken. blapi statically
links with softoken.. Perhaps we should move blapi-test to nss-softoken in the
future?

[reply] [-]
Private
Comment 18 Bob Relyea 2011-08-29 20:42:57 EDT

Comment on attachment 520484 [details]
spec file changes to build without freebl/softoken/util sources in the tree

r+ for now, but we aren't complete until we straighten out the header file
situation.

[reply] [-]
Private
Comment 19 Elio Maldonado Batiz 2011-09-08 19:33:31 EDT

(In reply to comment #16)
I was able to copy the system token into the dist via the spec file but that
caused the fips test to fail I tried it via fips.sh copying into the mangling
directory and had the same fips test failures.

[reply] [-]
Private
Comment 20 Elio Maldonado Batiz 2011-09-08 19:40:05 EDT

Misstyped: s/system token/softoken.so/

[reply] [-]
Private
Comment 21 Elio Maldonado Batiz 2011-09-08 19:46:45 EDT

Also, in Rawhide I am able to run the cipher suite (blapitest) as part of
nss-softoken build itslef. I had to do some modidifications to blapitest and
create a set of the libsectool.a that only depends on softoken and below. I use
for testing and don't install it in the system and is a temporary workaround.
Yes, it's a bit hackish and can't bring it into rhel 6 for nss-softoken,
certainly not at this stage. The changes in nss 3.13 will make things a lot
easier.
Comment 3 Elio Maldonado Batiz 2012-01-12 10:22:36 EST
Created attachment 552424 [details]
Main patch for building nss without softoken or util in the tree
Comment 4 Elio Maldonado Batiz 2012-01-12 10:24:47 EST
Created attachment 552425 [details]
Adds a target to the Makefile to copy prebuilt softoken libs to the tree
Comment 5 Elio Maldonado Batiz 2012-01-12 10:26:54 EST
Created attachment 552428 [details]
Copy the softoken libs from the buildroot to the build tree.
Comment 6 Elio Maldonado Batiz 2012-01-12 10:33:36 EST
Comment on attachment 552428 [details]
Copy the softoken libs from the buildroot to the build tree.

The tests are run against the shipping softoken. Also we can now run the mangling test.
Comment 7 Elio Maldonado Batiz 2012-01-12 10:39:46 EST
Created attachment 552434 [details]
If blapit.h lacks them this header provides them

This needed only for RHEL 6.x where blapit.h is from softokn is at 3.12.9 which lacks those defines. This is not needed for Fedora where everything is at 3.13.x.
Comment 8 Elio Maldonado Batiz 2012-01-12 10:46:08 EST
Created attachment 552436 [details]
Changes to the spec file

Applies the patches and copies headers that we need for building. In the future we will need less copying as the patches we submitted upstream get approved and we update.
Comment 9 Elio Maldonado Batiz 2012-01-12 10:54:49 EST
Created attachment 552443 [details]
Disable two stress that are currently failing

This patch is temporary. When I build in my system with either 'fedpkg local' or with 'rpmbuild --rebuild nss-srpm' all test pass. When I submit a scratch build or when I build with 'fedpkg mockbuild' the two tests fail. Will get rid of this patch as soon as I figure out what' causing it and have a fix.
Comment 10 Elio Maldonado Batiz 2012-01-12 11:00:17 EST
Comment on attachment 552434 [details]
If blapit.h lacks them this header provides them

As you suggested.
Comment 11 Elio Maldonado Batiz 2012-02-13 17:55:52 EST
Created attachment 561692 [details]
Disable ssl pkcs #11 bypass at build time

Break off this minimalist patch worth submitting upstream. Maintainer needs to define the NO_SSLBYPASS build variable, which I do in nss.spec.
Comment 12 Elio Maldonado Batiz 2012-02-13 17:58:12 EST
Created attachment 561694 [details]
Main patch to build nss without softoken or util sources in the tree
Comment 13 Elio Maldonado Batiz 2012-02-13 18:01:38 EST
Created attachment 561696 [details]
Adds a Makefile target to copy prebuild softoken list to build tree
Comment 14 Elio Maldonado Batiz 2012-02-13 18:03:18 EST
Created attachment 561697 [details]
Copies the nss-softoken shared lins to the build tree
Comment 15 Elio Maldonado Batiz 2012-02-13 18:05:32 EST
Created attachment 561698 [details]
Adds the missing defines blapit.h lacks on rhel-6.x with older softokn/freebl
Comment 16 Elio Maldonado Batiz 2012-02-13 18:14:30 EST
Created attachment 561699 [details]
Disable some ssl stress that are failing

This is temporary until I can find out why some ssl stress fail due apparently an expired certificate. All tests pass when I build via 'fedpkg build local' or via 'rmpbuild --rebuild path-to-srpm'. Failures occur with 'fedpkg mockbuild', and that's what's used on a Koji build. Will investigate and fix.
Comment 17 Elio Maldonado Batiz 2012-02-13 18:17:08 EST
Created attachment 561700 [details]
Changes to the spec file in patch form
Comment 18 Elio Maldonado Batiz 2012-02-13 18:22:37 EST
Comment on attachment 561696 [details]
Adds a Makefile target to copy prebuild softoken list to build tree

For some reason copying done from the spec, as would very much prefer, fails yet it works when I do via the target in the Makefile.
Comment 19 Elio Maldonado Batiz 2012-02-13 18:24:10 EST
Comment on attachment 561698 [details]
Adds the missing defines blapit.h lacks on rhel-6.x with older softokn/freebl

Needed for rhel-6 only.
Comment 20 Elio Maldonado Batiz 2012-02-13 18:25:32 EST
Comment on attachment 561699 [details]
Disable some ssl stress that are failing

Temporary until I fix things.
Comment 21 Bob Relyea 2012-02-14 13:48:53 EST
Comment on attachment 561692 [details]
Disable ssl pkcs #11 bypass at build time

r- 

This code doesn't even come close to disabling bypass.

1) it does not remove all the link type calls to freebl.

2) It still allows bypass to be set.

3. It still allows lots of dead code.

Examples of bypass code can be found in sssl3_initPendingcipherSpec, ssl3computeRechordMAC, ssl3RestartHandshakeHashes, ssl3NewHandshakeHashes, ssl3UpdateHandshakeHashes, (at this point I've determine I've made my point).

look for occurances of opt.bypassPKCS11.

Apparently you are still linking with freebl. You should remove freebl from the link line when NO_SSLBYPASS is set.
Comment 22 Bob Relyea 2012-02-14 13:56:23 EST
Comment on attachment 561694 [details]
Main patch to build nss without softoken or util sources in the tree

Question, what breaks when you don't make the following change:
 +ifndef USE_SYSTEM_FREEBL
 CRYPTOLIB=$(SOFTOKEN_LIB_DIR)/$(LIB_PREFIX)freebl.$(LIB_SUFFIX)
+SOFTOKENLIB=$(DIST)/lib/$(LIB_PREFIX)softokn.$(LIB_SUFFIX)
+else
+# Use the system one for freebl. For softokn on the other hand,
+# link with the system shared softoken library
+CRYPTOLIB=$(FREEBL_LIB_DIR)/$(LIB_PREFIX)freebl.$(LIB_SUFFIX)
+SOFTOKENLIB=
+EXTRA_SHARED_LIBS += \
+	-L$(SOFTOKEN_LIB_DIR) \
+	-lsoftokn3 \
+	$(NULL)
+endif

The code you are changing is the code that links with static libraries. It expects to link directly with the static libraries, you've changed it to dynamically link with softoken.

I see 3 things we can do here:

1) include a static version of softoken in the -devel library for softoken.

2) fix or remove the applications that are statically linking.

3) your patch as is.

I don't think option 3 if viable for upstream. We should at least investigate option 2. I'm tetatively r+, this but I would like info onw what nss tools break when we don't do this.

Also, We should make the -UNSS_ENABLE_ECC defines a separate environment variable. I can understand why it's necessary in the static link case, but you should be able to build with a system freebl that does have ECC.
Comment 23 Bob Relyea 2012-02-14 13:57:57 EST
Comment on attachment 561696 [details]
Adds a Makefile target to copy prebuild softoken list to build tree

I presume copy-nss-softoken-libs.sh is included somewhere in this review...
Comment 24 Bob Relyea 2012-02-14 14:00:32 EST
Comment on attachment 561697 [details]
Copies the nss-softoken shared lins to the build tree

r+ though, this is very linux specific. Also, is this necessary with our other changes?
Comment 25 Bob Relyea 2012-02-14 14:03:39 EST
Comment on attachment 561698 [details]
Adds the missing defines blapit.h lacks on rhel-6.x with older softokn/freebl

Is this only needed by libssl? Are we exporting this to other parts of NSS.

I'm OK with doing this if it's really needed, but I think fixing the ifdef NO_BYPASS should be able to fix with without the header.
Comment 26 Bob Relyea 2012-02-14 14:05:04 EST
Comment on attachment 561699 [details]
Disable some ssl stress that are failing

r- probably failing because they are testing session ticket code with bypass. I would expect this failures since you haven't finished disabling bypass.
Comment 27 Bob Relyea 2012-02-14 14:07:38 EST
Comment on attachment 561700 [details]
Changes to the spec file in patch form

r+, though you need to deal with the other things I commented on.

good documenentation on what high level functions include low level blapi headers.

bob
Comment 28 Bob Relyea 2012-03-13 18:24:55 EDT
Comment on attachment 561698 [details]
Adds the missing defines blapit.h lacks on rhel-6.x with older softokn/freebl

See comment 25 We can revisit this after the bypass ifdef change.
Comment 29 Elio Maldonado Batiz 2012-03-25 15:19:36 EDT
Created attachment 572565 [details]
Disable ssl bypass at build time

I thought this part was important enough to deserve its own bug. This is the attachment for Bug 806588.
Comment 30 Elio Maldonado Batiz 2012-03-30 18:40:34 EDT
Created attachment 574101 [details]
Disable ssl pkcs11 layer bypass at build time

The same patch attached to Bug 806588.
Comment 31 Elio Maldonado Batiz 2012-05-01 19:06:57 EDT
(In reply to comment #22)
> Comment on attachment 561694 [details]
> Main patch to build nss without softoken or util sources in the tree
> 
> Question, what breaks when you don't make the following change:
>  +ifndef USE_SYSTEM_FREEBL
>  CRYPTOLIB=$(SOFTOKEN_LIB_DIR)/$(LIB_PREFIX)freebl.$(LIB_SUFFIX)
> +SOFTOKENLIB=$(DIST)/lib/$(LIB_PREFIX)softokn.$(LIB_SUFFIX)
> +else
> +# Use the system one for freebl. For softokn on the other hand,
> +# link with the system shared softoken library
> +CRYPTOLIB=$(FREEBL_LIB_DIR)/$(LIB_PREFIX)freebl.$(LIB_SUFFIX)
> +SOFTOKENLIB=
> +EXTRA_SHARED_LIBS += \
> + -L$(SOFTOKEN_LIB_DIR) \
> + -lsoftokn3 \
> + $(NULL)
> +endif
> 
> The code you are changing is the code that links with static libraries. It
> expects to link directly with the static libraries, you've changed it to
> dynamically link with softoken.
> 
> I see 3 things we can do here:
> 
> 1) include a static version of softoken in the -devel library for softoken.
> 
> 2) fix or remove the applications that are statically linking.
> 
> 3) your patch as is.
> 
> I don't think option 3 if viable for upstream. We should at least investigate
> option 2. I'm tetatively r+, this but I would like info onw what nss tools
> break when we don't do this.

On option 2, the tools that break if I don't make this change (or one less intrusive) are blapitest, certcgi, rsaperf, and ocspclntst. They all have USE_STATIC=1 in their manifest.mn. 

blapitest is no problem as we now build it and run those tests as part of the nss-softoken package build were it more logically belongs.

2. certcgi is a tool that is not used by anything else. It's documentation mentions an internal test CA server inside Netscape. It looks like some historical tool. Could it be a candidate for removal?

3. rsaperf tests rsa performance via the PK11_ wrappers as well as via pkcs- #11 calls so it needs access to the static softoken library.

4. ocspclnt needs CERT_DecodeOCSPRequest, CERT_CheckOCSPStatus, and CERT_DecodeOCSPRequest from libcerthi.a.


> Also, We should make the -UNSS_ENABLE_ECC defines a separate environment
> variable. I can understand why it's necessary in the static link case, but you
> should be able to build with a system freebl that does have ECC.
I'll look this.
Comment 32 Bob Relyea 2012-05-01 20:20:49 EDT
> 2. certcgi is a tool that is not used by anything else. It's documentation
> mentions an internal test CA server inside Netscape. It looks like some
> historical tool. Could it be a candidate for removal?

yes, we should either fix it or remove it. I'm sure the current code is bitrotted (I think it predates NSS as a shared library).

> 3. rsaperf tests rsa performance via the PK11_ wrappers as well as via 
> pkcs-#11 calls so it needs access to the static softoken library.

rsaperf goes to freebl directly, not softoken. I think it would be a good idea to split rsaperf into high and low level utilities -- one that calls from the pkcs11 layer, and one that calls directly to freebl. Put the low level one in nss-softoken. It should be relatively straight forward.

> 4. ocspclnt needs CERT_DecodeOCSPRequest, CERT_CheckOCSPStatus, and
> CERT_DecodeOCSPRequest from libcerthi.a.

This one is the tricky one. It's calling 3 OCSP functions which are not exported. I wonder if we shouldn't just export those functions.

bob
Comment 33 Bob Relyea 2012-05-04 13:52:47 EDT
Comment on attachment 574101 [details]
Disable ssl pkcs11 layer bypass at build time

r+ with some caveats:

1) I'm sure there may be some upstream comments for this patch.

2) You have a few asserts for !bypass. I think you could remove them.

3) I don't see the SSL_OptionSet change, but I would prefer it just silently allowed 'bypass' to be set with no effect. From a semantic point of view the only app difference between bypass on or off is a timing issue. For RHEL at least we should allow apps that set bypass to quietly continue with the fiction. (what we do upstream, in fedora or RHEL 7 is a different matter).

In your first assert, you have 2 returns: Failure and try with bypass off. If you address my first comment the issue goes away. If you don't, you need to decide between these 2 options.

OK, it's pretty nuanced r+...;)
Comment 34 Elio Maldonado Batiz 2012-05-14 14:05:10 EDT
Comment on attachment 574101 [details]
Disable ssl pkcs11 layer bypass at build time

This patch is a bit old. I know have one that will not only disable bypass via the NSS_NO_PKCS11_BYPASS build time environment variable but also you the allow the package maintainer to specify that ABI compatibility be preserved.
Comment 35 Elio Maldonado Batiz 2012-05-14 14:13:55 EDT
Created attachment 584429 [details]
Disable ssl pkcs11 bypass at build time and preserve ABI
Comment 36 Elio Maldonado Batiz 2012-05-14 14:16:55 EDT
Comment on attachment 584429 [details]
Disable ssl pkcs11 bypass at build time and preserve ABI

Same as the patch I'm attaching to Bug 806588 and which I'm also sending upstream.
Comment 37 Elio Maldonado Batiz 2012-05-24 14:19:56 EDT
Comment on attachment 561699 [details]
Disable some ssl stress that are failing

The failed stress are no longer a problem with the latest version of the latest patch currently under review upstream
Comment 38 Elio Maldonado Batiz 2012-11-10 12:03:05 EST
Comment on attachment 584429 [details]
Disable ssl pkcs11 bypass at build time and preserve ABI

Patch no longer needed, implemented upstream for nss-3.14 and available with the update plus changes for Bug 806588.
Comment 39 Elio Maldonado Batiz 2012-11-10 17:46:03 EST
The patches submitted for Mozilla Foundation External trackers were accepted and applied upstream and icluded in the nss-3.14 release to which we have updated. Additionally, all of the previously r+'ed patches need updating on account of the nss-3.14 rebase.
Comment 40 Elio Maldonado Batiz 2012-11-11 22:50:05 EST
Created attachment 643175 [details]
Remove include.h from seckey.c is not needed

Patch I plan to submit upstream.
Comment 41 Elio Maldonado Batiz 2012-11-11 22:58:16 EST
Created attachment 643180 [details]
copies system softoken libraries into to {DIST}/lib in the source tree

This script is invoken from a new target in mozilla/security/nss/Makefile. Not the way I wanted but my attemps at doing the copy from the spec file itself have failed as I could get te destination directory to be well defined. From a target in the Makefile it works.
Comment 42 Elio Maldonado Batiz 2012-11-11 23:23:21 EST
Created attachment 643202 [details]
patches build system files to build nss with softoken or util in the tree

Cleaner that the previous version thanks inpart to the rebase to nss-3.14. This patch works on config.mk, manifest.msn, and Makefiles only. It doesn't touch sources. A separate one does that for in crypohi/seckey.c, will submit upstream.

The rsaperf tool is still problematic. Splitting it into high and level versions, as suggested, is harder than I first thought. The the low level partion  has some PK11_ calls that need be replaced. This will have to wait and when I do submit a patch upstream.
Comment 43 Elio Maldonado Batiz 2012-11-11 23:30:50 EST
Created attachment 643203 [details]
Changes to the spec file in patch form

This version does a lot less copying of headers thanks in great measure to the patches accepted and applied upstream for nss-3.14 plus some experimentation on my part.
Comment 44 Elio Maldonado Batiz 2012-11-11 23:33:36 EST
Comment on attachment 643175 [details]
Remove include.h from seckey.c is not needed

Should submit upstream.
Comment 45 Elio Maldonado Batiz 2012-11-11 23:36:12 EST
Comment on attachment 561700 [details]
Changes to the spec file in patch form

Obsoleted by the nss-3.14 based one.
Comment 46 Elio Maldonado Batiz 2012-11-12 13:32:09 EST
Created attachment 643660 [details]
Changes to the spec file in patch form
Comment 47 Elio Maldonado Batiz 2012-11-12 17:30:35 EST
Created attachment 643794 [details]
patch build system files to build without softoken or util in the tree

Another iteration, no need to patch anything in coreconf.
Comment 48 Elio Maldonado Batiz 2012-11-12 17:33:43 EST
Created attachment 643807 [details]
Changes to the spec file in patch form
Comment 49 Bob Relyea 2012-11-13 17:45:48 EST
Comment on attachment 643175 [details]
Remove include.h from seckey.c is not needed

r+ rrelyea. make sure it still build with ECC when you push this upstream.
Comment 50 Bob Relyea 2012-11-13 17:54:46 EST
Comment on attachment 643794 [details]
patch build system files to build without softoken or util in the tree

r-

Do you really need freebl to link NSS? NSS does not depend directly on freebl. I also don't think you need to link softoken either (NSS explicitly loads softoken, or it used to).

In ssl/config.mk, make linking with freebl conditional on BYPASS.

The shell script to copy softoken is missing. perhaps it's in the patch that looks like it is broken?

-----------test-----------------
Perhaps you could check to see if bltest is installed from bin/unsupported? (or maybe your softoken copies it from bin/unsupported, in which case this code is fine.
Comment 51 Bob Relyea 2012-11-13 17:56:01 EST
Comment on attachment 643180 [details]
copies system softoken libraries into to {DIST}/lib in the source tree

r+

You may also want to copy any binaries (like bltest) if they exist..
Comment 52 Bob Relyea 2012-11-13 18:53:39 EST
Comment on attachment 643807 [details]
Changes to the spec file in patch form

I'm giving this an r+, but I do have some reservations. There are things that need upstream bugs to get fixed.... In particular you shouldn't have to copy any headers around.


> # nssysinit.c needs pkcs11ni.h
> %{__cp} ./mozilla/security/nss/lib/softoken/pkcs11ni.h ./mozilla/security/nss/lib/pk11wrap/

We need to determine what nsssysinit.c needs in pkcs11ni.h, and move defines around appropriately (probably to pkcs11n.h)


> # This is understandable
> %{__cp} ./mozilla/security/nss/lib/util/templates.c ./mozilla/security/nss/lib/nss

Is it, you need to explain why...
	

> # TODO: when resubmitting pem upstream find a way for pem to stop using freebl
> # upstream: https://bugzilla.mozilla.org/show_bug.cgi?id=402712

libpem *CAN* use freebl. It's  a PKCS #11 module.
Comment 53 Elio Maldonado Batiz 2012-11-30 19:20:36 EST
(In reply to comment #52)
> Comment on attachment 643807 [details]
> Changes to the spec file in patch form
> 
> I'm giving this an r+, but I do have some reservations. There are things
> that need upstream bugs to get fixed.... In particular you shouldn't have to
> copy any headers around.

An r+ with reservations is reason enough for me to not commit and address them.

> > # nssysinit.c needs pkcs11ni.h
> > %{__cp} ./mozilla/security/nss/lib/softoken/pkcs11ni.h ./mozilla/security/nss/lib/pk11wrap/
> 
> We need to determine what nsssysinit.c needs in pkcs11ni.h, and move defines
> around appropriately (probably to pkcs11n.h)

It doesn't to include it will attach a patch for it not to which I got accepted and comitted upstream already for nss-3.14.1.

> > # This is understandable
> > %{__cp} ./mozilla/security/nss/lib/util/templates.c ./mozilla/security/nss/lib/nss
> 
> Is it, you need to explain why...
Oh, that was and old comment thaqt I forgot the remove. Indeed, it's not undertandable. This what I found. If I remove the include it builds fine, but only on fedora. I tried the same upstream where I have the full source tree and though it on fedora it failed on Mac OS X and Windows. I haven't found a good explanation yet.
	
> > # TODO: when resubmitting pem upstream find a way for pem to stop using freebl 

> > # upstream: https://bugzilla.mozilla.org/show_bug.cgi?id=402712
> libpem *CAN* use freebl. It's  a PKCS #11 module.

Yes. It crosssed my mind that a module should use the pkcs #11 api to have other modules do work it doesn't do itself. I had to do such things in previous projects  but that was because I had to comply with a provider archtecture. Glad to hear that now I don't to do that. 

I have a cleaner version that could attach. The templates.c issue needs explaining though.
Comment 54 Elio Maldonado Batiz 2012-12-27 17:42:45 EST
Created attachment 669709 [details]
patch build system files to build without softoken or util in the tree
Comment 55 Elio Maldonado Batiz 2012-12-27 17:48:53 EST
Created attachment 669710 [details]
split rsaperf into high and low level components

nss-3.14.1 backport of the patch submitted upstream for nss-3.14.2. rsaperf_low is built as part of the nss-softokn build.
Comment 56 Elio Maldonado Batiz 2012-12-27 17:56:26 EST
Created attachment 669711 [details]
adds /usr/includes/templates to the includes list

The templates directory was originally created so mod_revocator could "include" the nssck.api template and we reuse it. nss-util will install templates.c there.
Comment 57 Elio Maldonado Batiz 2012-12-27 17:59:17 EST
Created attachment 669712 [details]
Changes to the spec file in patch form
Comment 58 Elio Maldonado Batiz 2012-12-27 18:04:20 EST
Created attachment 669713 [details]
copy system softoken libraries and tools to dist in the tree
Comment 59 Elio Maldonado Batiz 2012-12-30 23:15:05 EST
This test from tests/cipher.sh is incorrect.

108 if [ ! -f ${DIST}/${OBJDIR}/bin/bltest -o  \
109 	 ! -f ${DIST}/${OBJDIR}/bin/bltest{PROG_SUFFIX} ]; then
110     echo "bltest not built, skipping this test." >> ${LOGFILE}

The '-o' should be '-a' and also the -f test is for regular files but we usually have symbolic links. Testing with -x will cover both regular files and symbolic links. It should be changed to

108 if [ ! -x ${DIST}/${OBJDIR}/bin/bltest -a  \
109 	 ! -x ${DIST}/${OBJDIR}/bin/bltest{PROG_SUFFIX} ]; then
110     echo "bltest not built, skipping this test." >> ${LOGFILE}

or simply
108 if [ ! -x ${DIST}/${OBJDIR}/bin/bltest{PROG_SUFFIX} ]; then
109     echo "bltest not built, skipping this test." >> ${LOGFILE}

I cautch this while using a version of this patch upstream where it's intended to change nothing on the build and the test suite of all.sh unless the proper environment variables are changed. A new patch is coming.
Comment 60 Elio Maldonado Batiz 2013-01-03 18:11:28 EST
Comment on attachment 669710 [details]
split rsaperf into high and low level components

Let's wait for the upstream review.
Comment 61 Elio Maldonado Batiz 2013-01-03 18:14:12 EST
Comment on attachment 669712 [details]
Changes to the spec file in patch form

Let's wait for the upstream review on the splitting rsaperf patch. The spec file will likelly need to be modified differently.
Comment 62 Elio Maldonado Batiz 2013-01-03 18:16:11 EST
Comment on attachment 669711 [details]
adds /usr/includes/templates to the includes list

Though this one is intended as a fedora-only patch it's best to wait for the upstream review of the other ones.
Comment 63 Elio Maldonado Batiz 2013-01-03 18:17:25 EST
Comment on attachment 669713 [details]
copy system softoken libraries and tools to dist in the tree

Lrt's wait for the upstream review.
Comment 64 Fedora End Of Life 2013-04-03 15:49:12 EDT
This bug appears to have been reported against 'rawhide' during the Fedora 19 development cycle.
Changing version to '19'.

(As we did not run this process for some time, it could affect also pre-Fedora 19 development
cycle bugs. We are very sorry. It will help us with cleanup during Fedora 19 End Of Life. Thank you.)

More information and reason for this action is here:
https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora19
Comment 65 Elio Maldonado Batiz 2013-06-18 17:22:44 EDT
Created attachment 762643 [details]
changes in patches and spec file to build nss without softoken

It takes advantage of the rebase to nss-3.15. All changes in one patch. I'll split it into separate patches if it turns out to be hard to view them.
Comment 66 Elio Maldonado Batiz 2013-06-18 17:37:22 EDT
Created attachment 762645 [details]
changes to spec file to build nss without softoken
Comment 67 Elio Maldonado Batiz 2013-06-18 17:40:06 EDT
Created attachment 762646 [details]
Pick up templates.c from the nss system templates directory
Comment 68 Bob Relyea 2013-06-18 18:42:19 EDT
OK, you still didn't explain templates.c. I had to go look.
Because of NSS ABI, we need to export certain template files from NSS. New NSS applications actually get the util version. I wonder if in Fedora we can drop nss/utilwrap.c altogether... Hmm... Well this is safer, so I'll r+ this.

bob
Comment 69 Bob Relyea 2013-06-18 18:42:50 EDT
Comment on attachment 762645 [details]
changes to spec file to build nss without softoken

r+ rrelyea
Comment 70 Bob Relyea 2013-06-18 18:43:15 EDT
Comment on attachment 762646 [details]
Pick up templates.c from the nss system templates directory

r+ rrelyea
Comment 71 Fedora Update System 2013-06-23 00:42:59 EDT
nss-softokn-3.15-3.fc19, nss-util-3.15-1.fc19, nss-3.15-5.fc19, nspr-4.10-2.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/FEDORA-2013-11174/nss-3.15-5.fc19,nss-softokn-3.15-3.fc19,nss-util-3.15-1.fc19,nspr-4.10-2.fc19
Comment 72 Fedora Update System 2013-06-23 15:03:18 EDT
Package nss-softokn-3.15-3.fc19, nss-util-3.15-1.fc19, nss-3.15-5.fc19, nspr-4.10-2.fc19:
* should fix your issue,
* was pushed to the Fedora 19 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing nss-softokn-3.15-3.fc19 nss-util-3.15-1.fc19 nss-3.15-5.fc19 nspr-4.10-2.fc19'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-11174/nss-3.15-5.fc19,nss-softokn-3.15-3.fc19,nss-util-3.15-1.fc19,nspr-4.10-2.fc19
then log in and leave karma (feedback).
Comment 73 Fedora Update System 2013-06-29 14:17:13 EDT
nss-softokn-3.15-3.fc19, nss-util-3.15-1.fc19, nss-3.15-5.fc19, nspr-4.10-2.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 74 Fedora Update System 2013-07-07 13:51:23 EDT
nss-3.15.1-1.fc19,nss-softokn-3.15.1-1.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/nss-3.15.1-1.fc19,nss-softokn-3.15.1-1.fc19
Comment 75 Fedora Update System 2013-07-16 23:09:05 EDT
nss-3.15.1-1.fc19, nss-softokn-3.15.1-1.fc19, nss-util-3.15.1-1.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 76 Fedora Update System 2013-07-23 11:08:55 EDT
nss-util-3.15.1-1.fc18,nss-softokn-3.15.1-1.fc18,nss-3.15.1-1.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/nss-util-3.15.1-1.fc18,nss-softokn-3.15.1-1.fc18,nss-3.15.1-1.fc18
Comment 77 Fedora Update System 2013-08-03 19:59:57 EDT
nss-util-3.15.1-1.fc18, nss-softokn-3.15.1-1.fc18, nss-3.15.1-1.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.