Bug 1659737 - imagefactory Python 3 plan?
Summary: imagefactory Python 3 plan?
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: imagefactory
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Ian McLeod
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: PY2FTBI F31FailsToInstall PY2REMOVAL 1702996
TreeView+ depends on / blocked
 
Reported: 2018-12-15 23:24 UTC by Miro Hrončok
Modified: 2019-06-24 21:51 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-06-24 21:51:56 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github redhat-imaging imagefactory issues 423 'None' 'open' 'Port to Python 3' 2019-11-12 19:49:39 UTC

Description Miro Hrončok 2018-12-15 23:24:53 UTC
In line with the Mass Python 2 Package Removal [0], the following (sub)packages of imagefactory-plugins were marked for removal:

 * imagefactory-plugins-Docker
 * imagefactory-plugins-GCE
 * imagefactory-plugins-HyperV
 * imagefactory-plugins-IndirectionCloud
 * imagefactory-plugins-OVA
 * imagefactory-plugins-RHEVM
 * imagefactory-plugins-TinMan
 * imagefactory-plugins-vSphere

According to our query, those (sub)packages only provide a Python 2 importable module. If this is not true, please tell us why, so we can fix our query.

Please remove them from your package.

As said in the change document, if there is no objection in a week, we will remove the package(s) as soon as we get to it. This change might not match your packaging style, so we'd prefer if you did the change. If you need more time, please let us know here.

If you do the change yourself, it would help us a lot by reducing the amount of packages we need to mass change.

We hope this doesn't come to you as a surprise. If you want to know our motivation for this, please read the change document [0].

[0] https://fedoraproject.org/wiki/Changes/Mass_Python_2_Package_Removal

Comment 1 Miro Hrončok 2019-01-17 11:25:12 UTC
Can we remove imagefactory from Fedora completely instead?

What is your Python 3 plan?

Comment 2 Miro Hrončok 2019-02-21 19:07:00 UTC
Please?

Comment 3 Miro Hrončok 2019-02-26 12:34:37 UTC
Consider this a nonresponsive BZ

This the first reminder.

https://docs.fedoraproject.org/en-US/fesco/Policy_for_nonresponsive_package_maintainers/

Comment 4 Miro Hrončok 2019-03-05 10:01:55 UTC
This the second reminder. Are you responsive?

https://docs.fedoraproject.org/en-US/fesco/Policy_for_nonresponsive_package_maintainers/

Comment 5 Miro Hrončok 2019-03-12 10:00:51 UTC
This the third reminder. Are you responsive?

https://docs.fedoraproject.org/en-US/fesco/Policy_for_nonresponsive_package_maintainers/

Comment 6 Peter Robinson 2019-03-12 10:20:22 UTC
(In reply to Miro Hrončok from comment #1)
> Can we remove imagefactory from Fedora completely instead?

We can't remove this,  it's used by rel-eng for arm/cloud/container images.

Comment 7 Miro Hrončok 2019-03-12 10:41:59 UTC
There was a second part of the question:

What is your Python 3 plan?

Comment 8 Miro Hrončok 2019-03-16 13:12:12 UTC
I've got an e-mail from Chris Lalancette on Tuesday:

> Sorry for not responding earlier.
> 
> I'm not really involved with imagefactory at all, though I am still the maintainer of Oz.
> As Daniel pointed out, you'll want to start with Ian Mcleod for that package (he can point you in the right direction).
> 
> 
> Chris 

Naturally I've asked Chris to give the package to Ian. So far, that did not happen, so I'm adding NEEDINFO Ian Mcleod.

What is imagefactory's Python 3 plan?

Note that Python 2 imagefactory dependencies are going away from Fedora https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/62H7QTOSNLFQNKQZWINSBMMIF4SDA7Q3/ - I assume that simply ignoring the Python 3 question won't work well given that "we can't remove this, it's used by releng".

Here's the upstream issue: https://github.com/redhat-imaging/imagefactory/issues/423 - oz is apparently Python 3 ready.

Comment 9 Ian McLeod 2019-03-17 18:59:39 UTC
Apologies folks.  My attention has been strongly focused

Comment 10 Ian McLeod 2019-03-17 19:38:36 UTC
Premature submission, let's try that again.

Apologies folks.  My attention has been strongly focused elsewhere for the last month.

Although it's had a happy home as part of koji, you'll see from the changelog that Image Factory is effectively in maintinence mode.

Because of it's role in rel-eng, Brendan Reilly (now copied on the bug) has very kindly stepped up and has been helping with some of these maintenance tasks.

The core feature set has not seen any change in years.  There's been a fair amount of bit-rot in other areas, particularly those related to actually communicating with external cloud providers.  These features are not used as part of the koji integration.

All this is to say that Python 3 migration will be the biggest change this project has seen in a long time.

I think the best approach would be to combine this with stripping out as many of the rotted components as possible.

What I'm aware is still being used as part of koji and perhaps in a few other pipelines:

imagefactory <--- Core library bits
imagefactory-plugins <--- Mostly metapackage
imagefactory-plugins-TinMan <--- Integration with Oz for basic disk image building (qcow2/raw outputs in koji)
imagefactory-plugins-Docker <--- Generation of filesystem tarballs and full Docker base images

These remaining packages are involved in the workflows to generate OVA files for vSphere and RHEV-M/oVirt/RHV as well as Vagrant Boxes for various providers.

imagefactory-plugins-RHEVM
imagefactory-plugins-vSphere
imagefactory-plugins-ovfcommon
imagefactory-plugins-OVA

As a _very_ first step, I'll spend a bit of time right now sorting through the rpm-level deps for these packages, determining their python 3 status and removing any that are truly dead (and cutting out the feature in the associated plugins).

Generally speaking, the things that are good candidates for culling:

The EC2-specific code - much of which dates back to the earlier style of "Disk image plus detached kernel and initrd" images that EC2 required before it supported HVM.  This seems to be very much a legacy feature of EC2 at this point and it's not something I'm aware of anyone using from Factory, certainly not in the rel-eng context.

The connectivity code for vSphere, OpenStack and RHEV-M - This is an area where supporting libraries have either fallen away or may not have migrated to Python3.  I don't believe any of this is widely used.

The REST-api daemon - I am aware of one user of this feature, Tim Flink.  I've CCed him on the bug and will follow up.

Comment 11 Miro Hrončok 2019-03-17 19:53:57 UTC
Thank You, Ian.

Comment 12 Ian McLeod 2019-03-26 15:16:36 UTC
A status update for those who may be watching this with any interest...

I have a work-in-progress branch for the python3 migration here:

https://github.com/redhat-imaging/imagefactory/tree/feature/python3_wip

At the time of this update, I've done basic sanity tests on:

* Base image generation via the CLI
* OVA/Vagrant generation via the CLI
* Non-https basic functionality of the REST interface

I have had to remove XML support from the REST code due the lack of a python3 capable "picklingtools".
I have confirmation from Tim Flink that he does use the daemon/REST functionality and I'm seeking
clarification/confirmation about whether this use case requires XML and/or SSL.

Finally, the upstream oz project, which is a core part of making Image Factory work, has landed python3 migration in F30:

https://bodhi.fedoraproject.org/updates/FEDORA-2019-9ac4c9713a

Image Factory attempts to import oz as a module (as opposed to running it via its CLI).  SO.... the above update breaks Factory in F30.

And that's fine.  What I'll likely do in the next week or so is just merge the python3 WIP stuff, build it and submit as an update.

It will be less broken than a python2.7 based Factory that attempts to import Oz.

Comment 13 Peter Robinson 2019-03-26 15:33:39 UTC
> And that's fine.  What I'll likely do in the next week or so is just merge
> the python3 WIP stuff, build it and submit as an update.

Can we land it into F31/rawhide first and verify things work there first. I put conditionals for py2/py3 into oz for F-30 because I feel we're too close to F-30 GA atm and Fedora infra relies heavily on this so I'd sooner let it bake in rawhide and coordinate py3 going back into F-30 once we know those use cases work and are stable.

Comment 14 Petr Viktorin 2019-03-26 15:55:10 UTC
For the Python 2 removal, backporting to F30 is not that important -- the focus is on Rawhide and future releases.

Comment 15 Miro Hrončok 2019-04-11 10:17:00 UTC
Ian, thanks for the update. Was there any progress since then?

Comment 16 Miro Hrončok 2019-04-23 09:54:30 UTC
Note that imagefactory doesn't install:

Error: 
 Problem: conflicting requests
  - nothing provides python2-libguestfs needed by imagefactory-1.1.11-2.fc30.noarch

Comment 17 Miro Hrončok 2019-04-29 20:06:24 UTC
A week has passed and this bug is still in the NEW state and the package does not install.

Please fix this or indicate that you are working ona  fix by setting the state to ASSIGNED.

After 3 such reminders, this package may be orphaned.

https://fedoraproject.org/wiki/Changes/F31_Mass_Python_2_Package_Removal#Removing_non-installable_packages_from_the_distro

Thanks

Comment 18 Miro Hrončok 2019-04-30 17:37:30 UTC
My previous comment might have come across a bit too aggressive. I'm sorry, that was not my intention.

In preparation for the Python 2 EOL, we are removing all non-installable Python 2 packages:
https://fedoraproject.org/wiki/Changes/F31_Mass_Python_2_Package_Removal#Removing_non-installable_packages_from_the_distro

I'm mass requesting information and I guess I was a bit too overzealous.

Note that you don't have to actually fix this right now, setting the bug to ASSIGNED will just mark this as being worked on, so I'll know it is being taken care of. If this happens to quickly, feel free to reach to me any time for help (with specific problems).

Comment 19 Miro Hrončok 2019-05-06 14:12:34 UTC
In preparation for the Python 2 EOL, we are removing all non-installable Python 2 packages:
https://fedoraproject.org/wiki/Changes/F31_Mass_Python_2_Package_Removal#Removing_non-installable_packages_from_the_distro

This bug is still in the NEW state and the package does not install. Please indicate you are working on a fix by setting the state to ASSIGNED. When this bug is four weeks in the NEW state, the package may be orphaned.

Note that you don't have to actually fix this right now, setting the bug to ASSIGNED will just mark this as being worked on, so I'll know it is being taken care of. If this happens too quickly, feel free to reach to me any time for help (with specific problems).

(My previous comment might have come across a bit too aggressive. I'm sorry, that was not my intention.)

(If you know for sure this package shall be removed, consider doing it.)

Thank You!

Comment 20 Miro Hrončok 2019-05-13 07:38:04 UTC
This bug is still in the NEW state and the package does not install. Please indicate you are working on a fix by setting the state to ASSIGNED. When this bug is four weeks in the NEW state, the package may be orphaned.

Comment 21 Miro Hrončok 2019-05-20 11:19:55 UTC
This bug is still in the NEW state and the package does not install. Please indicate you are working on a fix by setting the state to ASSIGNED. When this bug is four weeks in the NEW state, the package may be orphaned.

Comment 22 Peter Robinson 2019-05-20 12:03:26 UTC
It's being worked upon, it's also critical rel-eng infra

Comment 23 Miro Hrončok 2019-06-24 21:51:56 UTC
I'm just going to close this since it no longer bothers me (it is fixed from my POV). I'd appreciate if you could be a bit more responsive in bugzilla :(

Thanks for the fix.


Note You need to log in before you can comment on or make changes to this bug.