Making python-nss compatible with Python3 is a substantial effort. python-nss is not pure Python code where you can run the 2to3 conversion tool on Python code. Rather python-nss is written entirely in C using the Python2 C API.

It's something we know we need to do but there hasn't been any demand previously so it's been on the back burner. This is first actual request we received, but I suspect now that Python3 is becoming the preferred Python version we're going to start seeing a lot more requests of this nature in the near future.

I've CC'ed my manager Dmitri on this bz so he can weigh in on the request.

We are contemplating a major version upgrade of python-nss from it's current major version of zero (major version 0 was meant to imply changes would likely occur before a stable 1.0 release). The type of changes envisioned in progressing from 0 -> 1 would mostly be naming changes that would yield greater consistency in the API and group related entry points into namespaces (i.e. modules). For the application using python-nss this would likely mean running a conversion script on it's pure Python code to rename things (we would supply the conversion script).

I already know a conversion to Python3 will force some API changes (Python 3 uses different data types) so the probably the best course of action would be to combine the namespace clean up work with the Python 3 conversion so consumers of the library only take 1 conversion hit as opposed to two conversions.

Summary:

Both the existing namespace issues in version zero and supporting Python 3 will force API changes. We should do both simultaneously and this work needs to be scheduled.

Dmitri we need to discuss this work.

P.S.: We will try to keep the impact on applications minimal. At the moment I'm guessing the porting work for most developers to move to next version of python-nss would be measured in hours, not days.

Hope that helps.

John

Hi John,
thanks for the comprehensive answer!
I know that you can't convert a complicated piece of software such as python-nss by automatic tool, which is the reason why I opened this bug in the first place :)
Python 3 will definitely start getting more and more preferred, and eventually (though maybe in "distant" future) it will become Fedora's default.
Thanks for considering this and please let me know the result of your discussion when it's done.

This bug appears to have been reported against 'rawhide' during the Fedora 20 development cycle.
Changing version to '20'.

More information and reason for this action is here:
https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora20

Hi, we would like to use Python 3 on the default installation instead of Python 2 on Fedora 22.

From that perspective, your package is considered as IMPORTANT - that means, is has to be updated to Python 3, for our intention come true.

The goal here is, that at least for F22 you should provide python3- prefixed subpackage.

Please, help use update to Python 3 flawlessly.

Check if upstream already support Python 3, if yes, use it and add te support to the package.

If upstream doesn't support Python 3 yet, encourage it to do so by sending patches and offering your help.

When upstream is dead or unwilling to support Python 3, you'll need to patch this package on Fedora level. Try to avoid this as much as you can, but use it, if it's the last option.

Chances are, that you ARE the upstream. In that case, everything is easier, just do it yourself.

There is a table on wiki, that should list your package. Chances are, that you can see an upstream link that covers the problem. Anyway, please update the table with information you know.

https://fedoraproject.org/wiki/User:Churchyard/python3

I offer my help with this task, so if you have no idea, how to work on this, or it is just not your priority, don't hesitate to ask for help.

(As you've already realized, this is a bulk text, so if something is not quite exact about your package, sorry for that, just ask)

For people who may run across this via a web search (I note from John Dennis's comments that we are upstream so this whole clause isn't applicable here):

Note: do not follow this piece of advice as it is against the Packaging Guidelines: "When upstream is dead or unwilling to support Python 3, you'll need to patch this package on Fedora level. Try to avoid this as much as you can, but use it, if it's the last option."

If you are in this situation, you'll essentially be forking upstream in order to produce a python3 port.  In that situation, the proper thing to do is to create a new package with the python3 port.  It would be even better to create the proper upstream infrastructure as well (new upstream scm and issue tracker) but that isn't 100% required by the guidelines.

just another "me too": Most of my projects are still using Python 2.4-2.6 but a few months ago I started developing my first ("professional") project which uses Python 3 so having Python NSS would be really nice (especially given OpenSSL's "remarkable" code quality as demonstrated by Heartbleed).

When I ported other libraries (mostly using the six module) to Python 3 the end result was much better code as Python 3 forced me to think about Bytes vs. Strings.

Just FYI, I'm starting to work on this, so please don't duplicate my effort. I'll submit a patch here once it works. I'll be targeting compatibility with Python 2.6, 2.7 and >= 3.3.

re comment #8

Thank you for your offer of help. What are you working on and what are you planning on doing? I'm upstream for python-nss and I'm well aware of a number of issues and pitfalls that will have to be addressed. Plus I'll be reviewing any proposed patches. Let's coordinate, no sense in doing a lot of work if the approach taken isn't viable. The result should work with both Py2 and Py3 and will likely need to introduce API changes, hence the reason it's important to plan this out.

Also this work needs to be done upstream.

The state I want to reach is being able to pass all tests on both Python 2 and 3, while not changing behaviour of python-nss on Python 2 *at all*.

Generally, the approach I'm taking is using conditionally defined macros (depending on whether Python major version is "2" or "3 or above"). I think API changes will only be necessary on Python 3:
- PyInt will be replaced by PyLong, but that's fine, because PyLong is *the* integer type in Python 3 - no change for people using it from Python modules.
- PyUnicode should be used where it makes sense - most of the functions currently work just with PyString and IMO it generally makes sense to make them work with PyUnicode in Python 3. Other functions work both with PyString and PyUnicode in Python 2 - these can be converted to work with PyBytes and PyUnicode (and the code can in fact be same for 2 and 3 if bytesobject.h is included, which maps PyBytes* to PyString* in Python 2).
- Module initialization needs to be tinkered with, but again that's more about ifdefing it the right way so that it works on Python 3.

That's pretty much everything I've done so far. Assuming that you're ok with it, I'll continue in this direction and if I hit any conceptual problems, I'll reach out to you and discuss. When the patch is ready, should I submit it here or elsewhere?

Thanks!

Thanks for the info Slavek. I'd like to see work in progress patches just to make sure things are on the right track.

You didn't mention buffer objects. It's a concept in Python 2 that I don't think got carried over to Python 3, presummaby they've been replaced with Bytes.

Another thing we want to look at is the use of SecITEMS. An NSS SecITEM is really nothing other than a block of data. It has a count of octets and a buffer of octets. Originally the NSS developers wanted to include a tag that I identifed the type of data it was however that was never consistently done. I tried to make up for the lack of tagging by adding my own tag when I knew what the data block was for, but I've learned there are too many places where it's impossible to know so it was a fruitless exercise. I'm beginning to believe SecItem's should not exist as class in python-nss and instead it should really just be a Python Bytes object, i.e. just a block of octets.

Also you'll need to be very careful with input parameters defined as strings to make sure the intended usage is a character string vs. a block of octets. The same applies to Unicode <--> UTF-8 conversions which are done in a number of places.

Created attachment 921750 [details]
First rough attempt to make python-nss python3 compatbile

So, this is the first attempt at making python-nss python3 compatbile. Not everything is as it should be ATM, it's very rough in some places. Notes:
- There are several places that I need to check for correctness/where I'm unsure about the right behaviour in Python 3. All of them are marked with "TODOpy3", so you can easily search them in the patch.
- It builds with Python 3, although there are some warnings during compilation that I need to investigate (connected to the *_richcompare functions).
- I'm really unsure whether server/client should use Bytes for the data they return from _recv/_readline and other functions - I'm assuming yes.
- I'm using PyBytes for implementation of SecITEMs, as for dropping them and using just PyBytes instead - I'm unsure. It seems that they have some good aditions on top of PyBytes - maybe creating a subclass of PyBytes for this purpose would be better?

Right now, tests pass on both Python 2.7 and 3.3, everything should also work on Python 2.6, although I admit I haven't yet tried. I'll be grateful for all comments/thoughts/suggestions. Thanks!

John, any comments/concerns/questions about this patch? Thanks!

My appologies, I've been swamped trying to get another project out the door and haven't looked at the patch yet. I know it's on my to-do list and I promise I'll get to it soon.

John, any update?

I'm sorry, although I've invested considerable time into python-nss since the last time we spoke it wasn't in the area of Py3 porting, we needed several new features (version 0.17.0 is out for testing).

Here is the problem. I do this work as part of my job here at Red Hat. I have not been allocated the time to work on the Python 3 porting by my manager.

Discussions are on-going as to whether python-nss should be superseded by python-cryptography. The outcome of those discussions will impact how much time we continue to invest in python-nss. At the same time (actually just yesterday) discussions just arose as to whether we are prepared in Fedora to move to Py3, perhaps that what prompted your query because I mentioned these contributed patches. We are still in the middle of this discussion.

The reality is I have to be assigned this task before I can invest any significant time into this. If you want I can put you in touch with my manager.

Thanks for your response. Couple of questions:
- Which python-cryptography are we talking about, is it [1]?
- Assuming that nss is superseeded by python-cryptography, in which Fedora release will nss be obsoleted/retired?
- Assuming that nss is superseeded by python-cryptography, will nss be completely abandoned or will it still get some upstream support?

I'd really hate to fail the whole effort just because nss is not ported and other upstreams still use it in F22. Anaconda is one of nss users and I'm quite sure that Anaconda team would have hard time rewriting their code to use a different library, considering the amount of work they have with python3 porting.

Also, I invested few days of my work into the patch attached here, so I'd be really unhappy to see all that work go away. And frankly, I don't think there is much left to do after you accept my patch (at least I hope...).

If accepting this patch and testing it requires approval of your manager, then I'd certainly like to talk to him.


[1] https://pypi.python.org/pypi/cryptography/

I'd just like to let you know that this change shouldn't be needed for Fedora 22 as Blivet, the python library used by Anaconda, should use the new libblockdev library providing functionality that requires python-nss for Blivet now.

Sorry, I can't parse comment #19. Are you trying to say Annaconda no longer has a dependency on python-nss? Or that Blivet now has a dependency on python-nss? Or that there is no longer any dependency on python-nss with respect to the installer?

Anaconda's dependency on python-nss is wrong, it's blivet that actually requires python-nss and only in specific cases (that's why python-blivet doesn't have Requires: python-nss). However, blivet should now be ported to use the new libblockdev library that provides functionality for which blivet now requires python-nss.

In other words, F22 anaconda should be using blivet in a version that uses libblockdev and doesn't require python-nss.

I forgot to update this bug report, the port of python-nss to Py3 has been mostly completed. Here is an email I sent to participants but should have also been reflected here. Note the repository mentioned below is temporary until such time as the code is committed to the Mozilla repository.

------------------------------------------------------------------------------


Introduction
------------

The port of python-nss to Py3 is essentially done, it is
python-nss-1.0.0, here after referred to as v1.0 for brevity.

v1.0 for both Py2 and Py3 builds from the exact same source code.

The unit tests have been ported to Py3 and all the
unit tests work under both Py2 and Py3. The examples have not been
ported but the were working under Py2 with v1.0 so in theory once the
examples are ported they also should work under Py3 with v1.0.

The v1.0 port was extensive and touched a lot of code. To put things
in perspective python-nss consists of 4 independent Python extension
modules comprising over 30,000 lines of C code and well over 1,000
individual functions. This makes python-nss one of the largest, if not
the largest, Python extension module written. The v1.0 port modified
2,500 lines of code (600 individual diff hunks). It is inevitable with
that many changes made under time pressure mistakes will have been
made which are yet to be discovered.


Get the code
------------

python-nss is a Mozilla project, per Mozilla requirements the SCM is
mercurial (e.g. hg). I have not pushed my changes to Mozilla yet
because I think it's premature. However I've pushed my repo containing
v1.0 to fedorapeople. While we collaborate on the py3 port lets use
the repo on fedorapeople as a temporary area. You can get the source
with this:

hg clone ssh://jdennis@fedorapeople.org/~jdennis/public_html/python-nss/repo/python-nss

How to build and test
---------------------

Use the Python version of your choice, either Py2 (/usr/bin/python2)
or Py3 (/usr/bin/python3) to build and run programs.

For example to build under Py3:

% python3 setup.py build

To build under Py2:

% python2 setup.py build


If it's necessary to run under GDB to debug you can add the -d debug
option to the build command, that will turn on debug symbols and turn
off optimization:

% python3 setup.py build -d

You will get warnings concering FORTIFY_SOURCE with -d, it's a
non-issue.

To test I do not install, rather I set PYTHONPATH to the directory
under $topdir/build, e.g $topdir/build/lib.linux-x86_64-3.4, you may
need to adjust this to match your environment.

To run a script using python-nss simply invoke it with the desired
vesion of the Python interpreter, e.g.

% cd $topdir/test
% python3 setup_certs.py
% python3 test_cert_components.py

Strategy
--------

The following goals drove the porting effort:

* One common code base for both Py2 and Py3

* Support Python 2.7 and 3.x only (i.e. no versions earlier than
  Python 2.7)

* Code to the Python 3.x API and semantics. Python 3 is the current
  and future so the code "walks and talks" like Python 3.

* Minimize to the greatest extent possible macros and conditional
  compilation needed for Py2. The code should read like clean Py3
  code. Excessive use of macros and conditional compilation only serve
  to obfuscate the code and make it difficult to maintain.

* Properly support all encodings (e.g. use the idna codec on
  hostnames, filesystem code on pathnames, etc.).

Ramifications for Py2
---------------------

When built for Py2 python-nss honors the byte vs. text distinction
introduced in Py3 as well as the int/long changes.

Properly coded Py2 applications using v1.0 should have no
problems. IPA is an example of a properly coded application that uses
unicode for text and str for bytes.

Py2 applications which were not careful will probably be unaffected
because you can continue to pass in either str, utf-8, or unicode into
the python-nss API and the appropriate conversion will take place
internally. However in Py2 return values from v1.0 will be unicode for
text and str for bytes. In essence this means returned text strings
are *all* unicode in Py2. That shouldn't be a problem as Py2
seamlessly handles mixing str and unicode. The only place a
non-careful Py2 application using v1.0 might have a problem is if they
were explicitly testing for certain types being returned, something
that is not likely. The vast majority of text returned by python-nss
is informational; only used to print out the values and states of
objects.

The major version of python-nss has been bumped from 0 to 1 consistent
with API change. But there really isn't an API change of significance
if the python-nss user hasn't abused text and binary data types. Py2
v1.0 makes a good porting stage for Py3 because you're still running
under Py2 but python-nss acts as if it's Py3 but is forgiving of Py2
datatypes.

I would like to thank Bohuslav for the work he did in preparing an initial patch. While I wasn't able to directly apply the patch it did form the basis of many of the changes and was a tremendous help in getting the work done. I made constant reference to Bohuslav's work.

Hi John, do you plan to update and build python-nss with python3 support for Fedora 23+? It would be nice to have it before the Beta Freeze.

Thank you!

built python-nss-1.0.0-beta1.1.fc23

Can we close this?