Bug 1473433 - DNF API Always Uses Local Repository File
DNF API Always Uses Local Repository File
Status: CLOSED RAWHIDE
Product: Fedora
Classification: Fedora
Component: dnf (Show other bugs)
25
All Linux
unspecified Severity medium
: ---
: ---
Assigned To: Jaroslav Mracek
Fedora Extras Quality Assurance
: Reopened
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-07-20 16:16 EDT by rmarshall
Modified: 2017-10-18 08:14 EDT (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-07-22 04:35:06 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
a simple test of the DNF API (1.17 KB, text/x-python)
2017-07-20 16:16 EDT, rmarshall
no flags Details
Fedora-Rawhide-20171018.n.0/logs/global/traceback.global.log (17.21 KB, text/plain)
2017-10-18 07:22 EDT, Sergey Avseyev
no flags Details

  None (edit)
Description rmarshall 2017-07-20 16:16:29 EDT
Created attachment 1301985 [details]
a simple test of the DNF API

Description of problem:
While attempting to port some of Fedora's release engineering scripts from YUM to DNF discovered that it seems not possible to simply tell the DNF API to use only the repository specified and not the local package information.

Version-Release number of selected component (if applicable):


How reproducible:
See attached script running in Fedora 25 with latest DNF for python 3.5


Steps to Reproduce:
1. run test_dnf.py from the shell

Actual results:
/tmp/tmppf1c8hog-critpath
<function MultiCallList.__getattr__.<locals>.fn at 0x7fa90a2e38c8>
Group 'Core' is already installed, skipping.
Traceback (most recent call last):
  File "./test_dnf.py", line 38, in <module>
    for package in base.transaction.install_set:
AttributeError: 'NoneType' object has no attribute 'install_set'


Expected results:
I should not see that Group 'Core' is already installed; I cleared all local information and am using the upstream repo to just get a list of the packages that should be installed.

Showed this code to Ignore Gnatenko and he believed it to be a bug and asked me to file it in Bugzilla.
Comment 1 Jaroslav Mracek 2017-07-21 13:12:51 EDT
Please what version of dnf did you used?
Comment 2 Jaroslav Mracek 2017-07-21 13:17:52 EDT
There is a bug in your script . After 

``base.group_install('core', 'mandatory')``

you have to run

``base.resolve()``

before

``base.transaction.install_set`` is called

Please if it doesn't help, reopen the bug report.
Comment 3 rmarshall 2017-07-21 16:28:58 EDT
Re-opening with new information:

base.resolve() fixed the empty transaction issue in the isolated VM I created to test as well as my laptop; the issue continued on my desktop.

The 'Core' group was still marked as installed on my desktop machine. Investigating further, it seems that my desktop is different from the other two systems in that the file /var/lib/dnf/groups.json exists there.

-----OUTPUT FROM CODE-----
/tmp/tmpcem334r6-critpath
/tmp/tmpsng_h76p-critpath-installroot
<function MultiCallList.__getattr__.<locals>.fn at 0x7fc843ab7950>
Group 'Core' is already installed, skipping.
-----END CODE OUTPUT-----

The file is provided by the dnf-conf package according to 'dnf provides <path>'

Talked to Igor again - for now using this as a workaround in my code:

conf.persistdir = temp_cache_dir (same as my conf.cachedir)

Reading up on the docs, it looks like persistdir is used for keeping track of state between DNF runs and is not included with:

    base.reset(repos=True, goal=True, sack=True)

Igor asked me to re-open with this new information though, after reading the documentation on persistdir, it seems to me like the workaround is the desired method for running DNF outside of its normal usage in actually installing packages.

My use case is depsolving and getting just the list of packages for Fedora releng work in the critpath solver script among others.
Comment 4 Jaroslav Mracek 2017-07-22 04:35:06 EDT
Please, use dnf-2.5+ from copr repo (dnf copr enable rpmsoftwaremanagement/dnf-nightly). In dnf-1.1 there is a bug were even without successful install of group, the group was marked as installed.

Additional tips how to use an API can be found here https://dnf.readthedocs.io/en/latest/use_cases.html#extension-api

To unmark group as installed, please use ``dnf group mark remove <group-spec>...`` command.

Have a nice day.
Comment 5 Sergey Avseyev 2017-10-18 07:22 EDT
Created attachment 1340125 [details]
Fedora-Rawhide-20171018.n.0/logs/global/traceback.global.log

Similar issue currently reproducing on koji when building rawhide:

https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20171018.n.0/logs/global/traceback.global.log

https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20171018.n.0/logs/x86_64/buildinstall-Cloud.x86_64.log

2017-10-18 07:30:14,353: Performing post-installation setup tasks
Performing post-installation setup tasks
Fatal: no entropy gathering module detected
2017-10-18 07:31:20,635: writing .buildstamp file
writing .buildstamp file
2017-10-18 07:31:23,845: doing post-install configuration
doing post-install configuration
2017-10-18 07:31:23,864: running runtime-postinstall.tmpl
running runtime-postinstall.tmpl
2017-10-18 07:31:24,566: writing .discinfo file
writing .discinfo file
2017-10-18 07:31:24,594: backing up installroot
backing up installroot
2017-10-18 07:31:25,570: generating kernel module metadata
generating kernel module metadata
2017-10-18 07:31:25,570: doing depmod and module-info for 4.14.0-0.rc5.git1.1.fc28.x86_64
doing depmod and module-info for 4.14.0-0.rc5.git1.1.fc28.x86_64
2017-10-18 07:31:29,378: cleaning unneeded files
cleaning unneeded files
2017-10-18 07:31:29,438: running runtime-cleanup.tmpl
running runtime-cleanup.tmpl
2017-10-18 07:31:36,842: verifying the installroot
verifying the installroot
2017-10-18 07:31:41,965: creating the runtime image
creating the runtime image
Traceback (most recent call last):
  File "/usr/sbin/lorax", line 279, in <module>
    main()
  File "/usr/sbin/lorax", line 135, in main
    remove_temp=True, verify=opts.verify)
  File "/usr/lib/python3.6/site-packages/pylorax/__init__.py", line 331, in run
    rb.finished()
  File "/usr/lib/python3.6/site-packages/pylorax/treebuilder.py", line 235, in finished
    self.dbo.close()
  File "/usr/lib/python3.6/site-packages/dnf/base.py", line 415, in close
    self._finalize_base()
  File "/usr/lib/python3.6/site-packages/dnf/base.py", line 384, in _finalize_base
    if self._transaction.install_set:
AttributeError: 'NoneType' object has no attribute 'install_set'
Comment 6 Igor Gnatenko 2017-10-18 08:14:33 EDT
Sergey, this is different issue.

Note You need to log in before you can comment on or make changes to this bug.