Bug 1529071 - python3 removal leaves files behind
Summary: python3 removal leaves files behind
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Fedora
Classification: Fedora
Component: python3
Version: 27
Hardware: All
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Lumír Balhar
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-12-26 09:17 UTC by Artem S. Tashkinov
Modified: 2018-09-24 13:10 UTC (History)
15 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-09-24 13:10:11 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
directories not owned by package (4.14 KB, text/plain)
2018-01-03 08:17 UTC, Lumír Balhar
no flags Details

Description Artem S. Tashkinov 2017-12-26 09:17:14 UTC
$ ls -R /usr/lib/python3.4/
/usr/lib/python3.4/:
site-packages

/usr/lib/python3.4/site-packages:
dnf-plugins  seobject

/usr/lib/python3.4/site-packages/dnf-plugins:
__pycache__

/usr/lib/python3.4/site-packages/dnf-plugins/__pycache__:
builddep.cpython-34.pyc        debuginfo-install.cpython-34.pyc          needs_restarting.cpython-34.pyc    repoquery.cpython-34.pyc
config_manager.cpython-34.pyc  download.cpython-34.pyc                   noroot.cpython-34.pyc              reposync.cpython-34.pyc
copr.cpython-34.pyc            generate_completion_cache.cpython-34.pyc  protected_packages.cpython-34.pyc

/usr/lib/python3.4/site-packages/seobject:
__pycache__

/usr/lib/python3.4/site-packages/seobject/__pycache__:

$ ls -R /usr/lib/python3.5/
/usr/lib/python3.5/:
site-packages

/usr/lib/python3.5/site-packages:
dnf-plugins  isc

/usr/lib/python3.5/site-packages/dnf-plugins:
__pycache__

/usr/lib/python3.5/site-packages/dnf-plugins/__pycache__:
builddep.cpython-35.pyc        debuginfo-install.cpython-35.pyc          needs_restarting.cpython-35.pyc    repoquery.cpython-35.pyc
config_manager.cpython-35.pyc  download.cpython-35.pyc                   noroot.cpython-35.pyc              reposync.cpython-35.pyc
copr.cpython-35.pyc            generate_completion_cache.cpython-35.pyc  protected_packages.cpython-35.pyc

/usr/lib/python3.5/site-packages/isc:
__pycache__

/usr/lib/python3.5/site-packages/isc/__pycache__:

Comment 1 Miro Hrončok 2017-12-26 13:49:38 UTC
This is usually because some packages create files there that are not owned by those packages.

See for example here:



$ rpm -ql python3-bind
/usr/lib/python3.6/site-packages/isc-2.0-py3.6.egg-info
/usr/lib/python3.6/site-packages/isc/__init__.py
/usr/lib/python3.6/site-packages/isc/__pycache__/__init__.cpython-36.opt-1.pyc
/usr/lib/python3.6/site-packages/isc/__pycache__/__init__.cpython-36.pyc
...
/usr/lib/python3.6/site-packages/isc/__pycache__/utils.cpython-36.opt-1.pyc
/usr/lib/python3.6/site-packages/isc/__pycache__/utils.cpython-36.pyc
/usr/lib/python3.6/site-packages/isc/checkds.py
...
/usr/lib/python3.6/site-packages/isc/utils.py


The package does not own either /usr/lib/python3.6/site-packages/isc or /usr/lib/python3.6/site-packages/isc/__pycache__.

Now this is an error and it has to be fixed, but there is probably no way to fix it for the existing 3.4 and 3.5 files, because that would require issuing an update to already EOLed Fedora versions.

Comment 2 Artem S. Tashkinov 2017-12-26 13:54:48 UTC
None of the files in my original bug report belong to any installed packages - I verified that using `rpm -qf`

I do understand this bug report is impossible to fix for previous deprecated Python releases but it makes sense to resolve it for future Python versions.

Comment 3 Miro Hrončok 2017-12-26 14:12:51 UTC
Those are some directories (installed on my system) with similar issues:

$ LANG=C.utf8 find /usr/lib/python3.6/site-packages/  -type d -exec rpm -qf {} \; | grep -v locale | grep "is not owned by any package"
file /usr/lib/python3.6/site-packages/sphinxcontrib is not owned by any package
file /usr/lib/python3.6/site-packages/nbconvert is not owned by any package
file /usr/lib/python3.6/site-packages/initial_setup is not owned by any package
file /usr/lib/python3.6/site-packages/initial_setup/gui is not owned by any package
file /usr/lib/python3.6/site-packages/pykickstart is not owned by any package
file /usr/lib/python3.6/site-packages/pykickstart/commands is not owned by any package
file /usr/lib/python3.6/site-packages/pykickstart/handlers is not owned by any package
file /usr/lib/python3.6/site-packages/isc is not owned by any package
file /usr/lib/python3.6/site-packages/isc/__pycache__ is not owned by any package
file /usr/lib/python3.6/site-packages/sepolicy is not owned by any package

$ LANG=C.utf8 find /usr/lib64/python3.6/site-packages/  -type d -exec rpm -qf {} \; | grep -v locale | grep "is not owned by any package"
file /usr/lib64/python3.6/site-packages/pyanaconda is not owned by any package
file /usr/lib64/python3.6/site-packages/pyanaconda/__pycache__ is not owned by any package
file /usr/lib64/python3.6/site-packages/pyanaconda/ui is not owned by any package
file /usr/lib64/python3.6/site-packages/pyanaconda/ui/tui is not owned by any package
file /usr/lib64/python3.6/site-packages/report is not owned by any package
file /usr/lib64/python3.6/site-packages/reportclient is not owned by any package


I think we might need to create a scrip that:

 * installs all python3 packages (to a container or mock, etc.)
 * does the above queries
 * figures what packages have files in those dirs

CCing Python SIG.

dnf-plugins pycs are now properly owned:

$ rpm -qf /usr/lib/python3.6/site-packages/dnf-plugins/__pycache__
python3-dnf-plugins-extras-common-2.0.3-1.fc27.noarch
python3-dnf-2.7.5-1.fc27.noarch

$ rpm -qf /usr/lib/python3.6/site-packages/dnf-plugins/__pycache__/*.pyc 
python3-dnf-plugins-core-2.1.5-1.fc27.noarch
...
python3-dnf-plugins-core-2.1.5-1.fc27.noarch
python3-dnf-plugin-system-upgrade-2.0.3-1.fc27.noarch
python3-dnf-plugin-system-upgrade-2.0.3-1.fc27.noarch

I don't seem to be able to figure out where seobject comes from.

Comment 4 Petr Viktorin (pviktori) 2018-01-02 11:50:49 UTC
Not sure that script would work – are the files created by just installing the RPM?

seobject is probably from SELinux, judging by the name.

Comment 5 Miro Hrončok 2018-01-02 11:57:26 UTC
(In reply to Petr Viktorin from comment #4)
> Not sure that script would work – are the files created by just installing
> the RPM?

Yes. It's a directory where the files owned by the package are, i.e. see here from nbconvert %files section (irrelevant parts removed):

    %files -n python2-%{pypi_name}
    %{python2_sitelib}/%{pypi_name}-%{version}-py?.?.egg-info
    %{python2_sitelib}/%{pypi_name}/*

    %files -n python3-%{pypi_name}
    %{python3_sitelib}/%{pypi_name}-%{version}-py?.?.egg-info
    %{python3_sitelib}/%{pypi_name}/*

The %{pythonX_sitelib}/%{pypi_name} directories are not owned by the packages, only their contents. RPM will create the directory when installing, but it will remain there when the package is removed.

Comment 6 Lumír Balhar 2018-01-02 12:09:51 UTC
I am installing all packages with python3 in the name and I'll provide a list of directories with the same problem.

Comment 7 Miro Hrončok 2018-01-02 12:14:20 UTC
For each such directory, you want to trigger rpm -qf for some file in it (i.e. for __init__.py or even for * if you sort and uniq the results). That will give you the actual package that should own the directory.

Comment 8 Lumír Balhar 2018-01-03 08:17:17 UTC
Created attachment 1376217 [details]
directories not owned by package

I finished the investigation.

Procedure:
$ find /usr/lib*/python3.6/site-packages/  -type d -exec rpm -qf {} \; | grep -v locale | grep "is not owned by any package" > not_owned.txt
$ for folder in `cat not_owned.txt | cut -d " " -f 2`; do echo -n "$folder - "; rpm -qf `find $folder -type f | head -n 1` ; done

The result of the second command is attached. It found 47 folders which need to be fixed.

I also fixed one of them and fill a PR: https://src.fedoraproject.org/rpms/python-flask-babel/pull-request/1

A question is whether we want to fix them manually, create a script to fix them automatically or just open bugs or send an email to package owners.

Comment 9 Zbigniew Jędrzejewski-Szmek 2018-01-03 09:46:13 UTC
I think that it'd be most efficient if you could file PRs for all of them. Just filing bugs is as much work, and with PRs we're almost done ;)

I'd try to just "edit" the list into a series of sed -i invocations to mangle the spec files.

Comment 10 Lumír Balhar 2018-01-04 09:24:57 UTC
I've fixed all 47 directories in 26 packages, scratch-built them, tested a new version in mock and filled PRs. Some of PRs are merged already. I've also discovered some FTBFS bugs, so I'll try to fix them or at least create a bugs.

Will we continue somehow and, for example, create some automatic check for this issue?

Comment 11 Miro Hrončok 2018-01-04 11:24:54 UTC
taskotron check that tests directory not owned by any package makes sense (it is not only Python related). maybe put it in rpmgrill (or not)? talk to Kamil Páral or Martin Křížek, they might have some idea where to implement this. CC'ed.

Comment 12 Kamil Páral 2018-01-05 09:17:17 UTC
I think this check could be a good fit for either rpmlint or rpmgrill. I can't really tell which project is better for this, talk to the project owners or look how they're structured internally (I don't know.) We already execute these tasks in Taskotron, so finding out offending packages for this particular problem might be easy in the future (because some package maintainers will ignore the warnings).

Comment 13 Petr Viktorin (pviktori) 2018-01-18 13:43:56 UTC
Lumír, what's the status? Do you need any help?

Comment 14 Lumír Balhar 2018-01-18 15:25:00 UTC
There are still around 10 PR waiting for review and merge. We can wait one more week and then some proven packager could merge them.

I didn't have enough time to take a look at rpmlint/rpmgrill but I'm going to do it. Does it have a high priority?

Comment 15 Petr Viktorin (pviktori) 2018-01-18 15:35:02 UTC
No, it's definitely not high priority. Just checking the status.

Comment 16 Lumír Balhar 2018-04-10 13:48:42 UTC
I've implemented the check in the rpmlint tool: https://github.com/rpm-software-management/rpmlint/pull/131

If/when it'll appear in upstream, I can modify taskotron to install packages before rpmlint check.

Comment 17 Lumír Balhar 2018-05-14 17:12:54 UTC
As you can see in the mentioned pull request, after some discussion the proposed check for directory ownership was rejected. The reason is that tests of rpmlint itself don't count with testing installed RPMs and it seems that nobody uses this feature even is available and supported.

Also, there are no taskotron checks which check installed RPMs.

I see two possible ways how to do it:

1. Implement the same functionality directly in taskotron. It would be quite simple but the problem here is that taskotron is running checks on a minimal system and therefore installing RPM and its dependencies before every check could take a lot of time. Kamil Páral warned me that this check could be disabled if it would consume too many resources.

2. It might be possible to implement the same check without installing RPM package with just checking a package content but I am afraid that it can be really complex and we might end with the same issues mentioned in point 1.

A decision has to be made here and I am not sure that it worth to spend a lot of time on it.

[0] https://github.com/rpm-software-management/rpmlint/pull/131

Comment 18 Miro Hrončok 2018-05-14 17:23:12 UTC
Note that Fedora-Review runs check on installed RPMS. On the other hand I think it already has this check.

Comment 19 Petr Viktorin (pviktori) 2018-09-24 12:39:42 UTC
Lumír, I think we can't co anything here. Should we close it as WONTFIX or NOTABUG?

Comment 20 Lumír Balhar 2018-09-24 13:10:11 UTC
You are right, closing as WONTFIX.


Note You need to log in before you can comment on or make changes to this bug.