Hey there, Now that Azure has moved away from a big SDK bundle to multiple versioned packages, have you considered how to make that change with this package? I'm looking to package the Azure CLI but it has requirements for specific versions of SDK components. Thanks! Major
For what it's worth, I'm working through a proof of concept packaging scheme where all of the SDK components are included as sub-packages off a main python-azure package: https://github.com/major/rpm-azure-cli The spec file is generated based on the latest available GA version published by Azure. Would you like to collaborate on the packaging?
Hello Major, thanks for your interest in the Azure Python stack and the packaging scheme proposal. I've never been really satisfied with the current way the Azure SDK is packaged. Microsoft used to provide tagged releases for the SDK bundle (as requested by the Debian team 5 years ago IIRW) but it's no longer the case since last year/release 5.0.0. I was about to switch to snapshot releases, as Debian did, to keep the SDK updated... And (at last!) provide azure-cli in Fedora. So your proposal is very timely. I agree with splitting the SDK into dedicated libraries with right versioning. This is more satisfying. I'm also OK with the idea of keeping a single .spec file: it make things easier with your spec-generation script, and all Azure libraries are developed in a single Github repository after all. The choice is consistent. Using the python-packages.csv provided in the SDK sources to automate the spec file generation is a great idea BTW. I know the script is in working progress. Just a few questions/comments: - Do you plan to create a new python-azure package? If it can make things easier, I can add you as comaintainer for the current python-azure-sdk package. I know the current package name doesn't match exactly the (obsolete) pypi package name, the spec naming was based on the repo name when I started working on the SDK. - If you plan to create a new package, don't forget to manage transition to the python-azure-sdk package, by adding all the Requires to each subpackage.
(In reply to Mohamed El Morabity from comment #2) > - If you plan to create a new package, don't forget to manage transition to > the python-azure-sdk package... s/ to / from /
Thanks for the reply, Mohamed! After digging around in the Debian package[0] for ideas and starting some conversations in the Azure development community, it might make sense to carry on with an approach similar to what you have now. Building individual RPMs for components does work, but the challenge comes when you want to test. Some modules have no tests shipped to pypi and Azure's own docs say that testing is really only meant to be done with a full checkout of the repository. 🤦🏻♂️ I'm wondering if we can adjust your existing spec to do a git checkout, exclude certain modules (like *-nspkg), and build everything else. If you'd be okay with me co-maintaining the package, I am happy to help. This falls into the responsibilities of my $dayjob. 😉 [0] https://tracker.debian.org/pkg/python-azure
(In reply to Major Hayden from comment #4) > Building individual RPMs for components does work, but the challenge comes > when you want to test. Some modules have no tests shipped to pypi and > Azure's own docs say that testing is really only meant to be done with a > full checkout of the repository. 🤦🏻♂️ Maybe there's a way to conciliate the package splitting and the build/test from the SDK sources. This can be done later, let's keep this good idea for a next iteration ;). > > I'm wondering if we can adjust your existing spec to do a git checkout, > exclude certain modules (like *-nspkg), and build everything else. The *-nspkg used to be required to build/package the *storage* libraries, when they where maintained in a separate repository. This is no longer the case today, they can go away. > If you'd be okay with me co-maintaining the package, I am happy to help. > This falls into the responsibilities of my $dayjob. 😉 Sure, i've just added you. Let me know if I did it right
Perfect! Thanks, Mohamed! I'll keep working on this spec and make a PR in Pagure. We can talk over the change there and see if we like it. 🌅
So after a bunch of research, I see a few options: 1) Package via commit from github.com/azure/azure-sdk-for-python The benefit here is that there's one download and one big package. However, some of the packages have conflicting dependencies that need to be handled carefully and/or patched to ensure they can install properly. 2) Package via pypi in subpackages under a big main spec file This still allows for a single RPM build process that results in lots of little subpackages. However, lots of the subpackages have odd little packaging quirks that must be handled individually. 3) Package each component separately While this might be tedious to get started, it should be a bit easier to manage over time since some of the SDK modules don't change often and they won't require frequent attention. Each dependency and packaging quirk could be handled independently. ------------------------------ No matter how we package it, the tests are likely not going to work because many depend on network access and many depend on external resources outside the module itself. Also, we will need to deal with the situation where a particular package, such as azure-mgmt-portal has a version of 1.0.0 in pypi but the current python-azure-sdk package in Fedora is at 5.0.0. I feel like #3 is a better idea since we have more control over how all of the pieces fit together, but it will be painful at first. What do you think?
I agree with #3 too. The amount of package reviews to come may be discouraging indeed... But nothing is impossible with a good dependency tree, a little bit of pyp2rpm, another one of tweaking and a lot of patience :D. I'd just like to keep python-azure-sdk as an empty package requiring all the planned subpackages, for convenience and upgrades.
(In reply to Mohamed El Morabity from comment #8) > I agree with #3 too. The amount of package reviews to come may be > discouraging indeed... But nothing is impossible with a good dependency > tree, a little bit of pyp2rpm, another one of tweaking and a lot of patience > :D. > I'd just like to keep python-azure-sdk as an empty package requiring all the > planned subpackages, for convenience and upgrades. Same here. Thanks for the reassurance. 🤗
Let's take python-azure-core for an example. Its latest version is 1.14.0, but python-azure-sdk is versioned at 5.0.0. How do we sort these appropriately without causing confusion? This is a new territory for me in managing RPMs. I've done some basic work with "Obsoletes" before, but I'd take any guidance anyone has.
Each submodule bundled in the python-azure-sdk has a corresponding Provides field: $ LANG=C dnf repoquery --provides python3-azure-sdk | grep azure-core Last metadata expiration check: 1:33:29 ago on Thu Jun 3 09:33:14 2021. python3.9dist(azure-core) = 1.4.1 python3.9dist(azure-core-tracing-opencensus) = 1~b5 python3.9dist(azure-core-tracing-opentelemetry) = 1~b4 python3dist(azure-core) = 1.4.1 python3dist(azure-core-tracing-opencensus) = 1~b5 python3dist(azure-core-tracing-opentelemetry) = 1~b4 The azure-core module, once packaged separately, must have the same py3dist Provides (this shouldn't be an issue thanks to the automatic Python dependency/provides generator). Packaged version can reasonnably supposed >= to the current one in the python-azure-sdk package (otherwise this may be an Azure issue ^^). This would be the same for all other modules. As for the python-azure-sdk package version, once all submodules packaged, I'd suggest resetting the version to e.g. "1.0.0", with a explicite Epoch. Since this package won't rely anymore on a tag release on the Azure SDK side, I guess we're free to manage version for a empty package, provided for smooth upgrades and convenience, as we want (only Requires on submodules). Version would be updated if a new module appears on the SDK => new Requires in this package. BTW there is currently one package in Rawhide depending on python3-azure-sdk: fence-agents-azure-arm (direct Requires on python3-azure-sdk).
Your suggestions make sense, Mohamed. Thanks for detailing that out. 👏🏻 For what it's worth, I'm hopefully getting to a good spot here: https://github.com/major/rpm-azure-cli https://copr.fedorainfracloud.org/coprs/mhayden/azure-cli/monitor/
I'm working through the reviews on the pre-requisites for the Azure SDK/CLI and they seem to be moving along well (see dependent tickets attached to this one). Once those are done, there are a bunch of spec files ready[0]. However, I'd like to know if I'm handling "Obsoletes" correctly there. As an example, the spec for python-azure-mgmt-keyvault[1] has an Obsoletes for versions of python3-azure-sdk under 5.0.1, but I'm not sure if that's the best way to handle it. Some of the SDK packages have a version number under 1.0.0 (like 0.6.1), and dnf is often choosing to install python3-azure-sdk each time instead of the new packages that I'm building in the COPR repo. I'd love any feedback you have! [0] https://github.com/major/rpm-azure-cli/blob/main/specs/ [1] https://github.com/major/rpm-azure-cli/blob/main/specs/python-azure-mgmt-keyvault.spec [2] https://copr.fedorainfracloud.org/coprs/mhayden/azure-cli/
(In reply to Major Hayden 🤠 from comment #13) > I'm working through the reviews on the pre-requisites for the Azure SDK/CLI > and they seem to be moving along well (see dependent tickets attached to > this one). > > Once those are done, there are a bunch of spec files ready[0]. However, I'd > like to know if I'm handling "Obsoletes" correctly there. As an example, the > spec for python-azure-mgmt-keyvault[1] has an Obsoletes for versions of > python3-azure-sdk under 5.0.1, but I'm not sure if that's the best way to > handle it. Some of the SDK packages have a version number under 1.0.0 (like > 0.6.1), and dnf is often choosing to install python3-azure-sdk each time > instead of the new packages that I'm building in the COPR repo. So you finally want to get rid of the python-azure-sdk package? In such a case, Obsoletes in splitted packages are indicated. But I'm a little bit disappointed by this decision (see my comment #8). Let me add to your stack all the SDK dependencies I used to maintain for 5 years. I think you can handle them alone. Good luck!
I went back and re-read your comment #8 and that may be a better method to make this change gradually and avoid weird problems with obsoletes and file conflicts. This is my first attempt to replace bits of a big package with smaller packages, so I'm still trying to learn this process. The first package that conflicts with python-azure-sdk is python-azure-core, which I just put in bugzilla today. Should I remove the obsoletes/provides from that package and proceed? I guess I am trying to figure out how to package all of the small pieces and bring them in without disrupting and causing issues with the python-azure-sdk package. Any suggestions?
This package has changed maintainer in Fedora. Reassigning to the new maintainer of this component.
I've done lots of research, talked to at least 8-9 developers, and as noted in bug 1970619[0], the plan going forward is to: * Obsolete python-azure-sdk * Build all of the component packages in a side tag * Bring in all of the packages when they build and install properly [0] https://bugzilla.redhat.com/show_bug.cgi?id=1970619#c14
FEDORA-2021-8c76c0321b has been pushed to the Fedora 35 stable repository. If problem still persists, please make note of it in this bug report.