Bug 1476926

Summary: produce equivalent to yum.log (log of package installs, updates, and removes)
Product: [Fedora] Fedora Reporter: plaffiazho <plaffiazho2>
Component: rpmAssignee: Packaging Maintenance Team <packaging-team-maint>
Status: CLOSED RAWHIDE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rawhideCC: averma, awilliam, cb20777, david, drbasic6, igor.raits, jmracek, jrohel, justin.mccormick, jwakely, lordmetgod, mdomonko, mihai, nenad, packaging-team-maint, pmatilai, redhatbugs, rpm-software-management, vchepkov, yetoohappy
Target Milestone: ---Keywords: FutureFeature, Reopened, Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-11-13 09:24:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1355764    
Bug Blocks:    
Attachments:
Description Flags
A script to pull yum.log out of dnf none

Description plaffiazho 2017-07-31 20:30:11 UTC
yum produced a nice neat log file which simply listed the dates and times for each package added, updated, and removed. This is extremely helpful for many purposes, such as:

1) tracking down which update caused a problem (useful in case it's necessary to revert to the older version, as well as for quality of bug reports to upstream)
2) tracking down how packages added/removed correlate with e.g. system performance or other issues
3) undoing a transaction (e.g. knowing all packages and dependencies which were installed if it's decided you don't want the software after all, or vis versa (think remove-with-leaves))
4) after-the-fact documentation of what software was manually added to an installation, which may need to be reproduced when migrating to another machine

An example of each type of entry in yum.log:

Apr 06 03:19:09 Updated: yum-3.2.29-81.el6.centos.noarch
May 25 20:10:54 Erased: postfix
Jul 12 05:59:27 Installed: kernel-2.6.32-696.3.2.el6.x86_64

Could this please be added to dnf?

Comment 1 Daniel Mach 2017-09-27 11:51:55 UTC
I suppose your request was to add this functionality to DNF (reassigning the component from libdnf to dnf).
The feature exists already - there's dnf.log and also dnf history command.

Comment 2 plaffiazho 2017-09-28 19:23:28 UTC
have you looked at dnf.log? it is not analogous to yum.log ...

my current dnf.log has 12,578 lines -- mostly useless information like the following (this is only a sample):
2017-09-28T17:30:02Z INFO --- logging initialized ---
2017-09-28T17:30:02Z DDEBUG timer: config: 7 ms
2017-09-28T17:30:02Z DEBUG Loaded plugins: builddep, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, needs-restarting, playground, repoclosure, repograph, repomanage, reposync
2017-09-28T17:30:02Z DEBUG DNF version: 2.6.3
2017-09-28T17:30:02Z DDEBUG Command: dnf makecache timer 
2017-09-28T17:30:02Z DDEBUG Installroot: /
2017-09-28T17:30:02Z DDEBUG Releasever: 26
2017-09-28T17:30:02Z DEBUG cachedir: /var/cache/dnf
2017-09-28T17:30:02Z DDEBUG Base command: makecache
2017-09-28T17:30:02Z DDEBUG Extra commands: ['makecache', 'timer']
2017-09-28T17:30:02Z DEBUG Making cache files for all metadata files.
2017-09-28T17:30:02Z INFO Metadata cache refreshed recently.
2017-09-28T17:30:02Z DDEBUG Cleaning up.

But even if it didn't have all the useless information to dig through, it only contains information going back 4 days!

executing "dnf history" also doesn't output a summary analogous to yum.log, but instead you get stuff like:
ID     | Command line             | Date and time    | Action(s)      | Altered
-------------------------------------------------------------------------------
    87 |                          | 2017-09-26 22:39 | I, U           |   21   
    86 |                          | 2017-09-26 15:27 | Install        |    3   
    85 |                          | 2017-09-24 22:57 | Update         |   16   
    84 |                          | 2017-09-23 18:21 | Update         |   17

then there's dnf.rpm.log, which at least shows packages installed and removed (doesn't look like it shows packages updated?), but again it only goes back 4 days, and MOST of the entries are like follows:
2017-09-28T18:27:06Z INFO --- logging initialized ---
2017-09-28T18:30:02Z INFO --- logging initialized ---
2017-09-28T18:42:11Z INFO --- logging initialized ---
2017-09-28T18:57:28Z INFO --- logging initialized ---
2017-09-28T19:10:27Z INFO --- logging initialized ---
2017-09-28T19:12:32Z INFO --- logging initialized ---

I'm not aware of any way to get something like yum.log to meet any of it's use cases in a reasonable way. am i missing something?

Comment 3 Jaroslav Rohel 2017-10-13 05:30:14 UTC
From my point of view is dnf history command more comfortable for user than log file. 

>then there's dnf.rpm.log, which at least shows packages installed and removed (doesn't look like it shows packages updated?), but again it only goes back 4 days,

You can get list of installed/removed packages from history command too. Try to use "dnf history info <ID>" command where <ID> is id of transaction. Eg. "dnf history 85"

I don't know why your "Command line" column is blank. My example of real usage of history command:

I want view last 3 transactions
# dnf history list last..last-2
ID     | Command line             | Date and time    | Action(s)      | Altered
-------------------------------------------------------------------------------
   619 | history undo last        | 2017-10-13 06:33 | Erase          |    2   
   618 | install -y https://kojip | 2017-10-13 06:29 | Install        |    2   
   617 | upgrade /tmp/tito/x86_64 | 2017-10-11 07:58 | Update         |    5 EE

And now I want more info about transaction 618
# dnf history info 618
Transaction ID : 618
Begin time     : Fri Oct 13 06:29:21 2017
Begin rpmdb    : 2700:aca7f0c0e95d9cabe14ed0960d122d4430dc386d
End time       : Fri Oct 13 06:29:22 2017 (1 seconds)
End rpmdb      : 2702:0e2c92db8732ca8947f1136bf354f236f76f8ea1
User           : Jaroslav Rohel <jrohel>
Return-Code    : Success
Command Line   : install -y https://kojipkgs.fedoraproject.org//packages/dbus/1.11.16/4.fc27/x86_64/dbus-debugsource-1.11.16-4.fc27.x86_64.rpm https://kojipkgs.fedoraproject.org//packages/dbus/1.11.16/4.fc27/x86_64/dbus-debuginfo-1.11.16-4.fc27.x86_64.rpm
Transaction performed with:
    Installed     dnf-2.7.2-1.git.61.4812d55.fc26.noarch (unknown)
    Installed     rpm-4.13.0.1-7.fc26.x86_64             @updates
Packages Altered:
    Install dbus-debuginfo-1:1.11.16-4.fc27.x86_64   @@commandline
    Install dbus-debugsource-1:1.11.16-4.fc27.x86_64 @@commandline

Comment 4 plaffiazho 2017-10-14 21:52:40 UTC
> You can get list of installed/removed packages from history command too.
> Try to use "dnf history info <ID>" command where <ID> is id of transaction.
> Eg. "dnf history 85"

that requires going through one at a time ... for me currently for over 100 transactions (and this is still pretty early in the life of this install (indeed, for you it's over 600)) ... plus, it requires sifting through a lot more text to get the information I'm interested in which was presented in yum.log

> I don't know why your "Command line" column is blank.
> My example of real usage of history command:

the example I provided was also from real usage. my guess at why those transactions showed a blank command line columns is that they were transactions performed through the dnf api by yumex rather than through the dnf command line program. some of the entries show something in the command line column for me and some don't.

Comment 5 Jaroslav Rohel 2017-10-16 08:46:19 UTC
> that requires going through one at a time ... for me currently for over 100 transactions (and this is still pretty early in the life of this install (indeed, for you it's over 600)) ... plus, it requires sifting through a lot more text to get the information I'm interested in which was presented in yum.log

"for" and "grep" can help.
Example:
for i in {515..619} ; do dnf history info $i ; done | grep -E 'time|Install|Upgrade|Reinstall|Erase'

I admit it is not very user-friendly.
But I think that better solution will be to improve "history" command than changing log file.

Comment 6 Fedora End Of Life 2017-11-16 19:04:51 UTC
This message is a reminder that Fedora 25 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 25. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '25'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 25 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

Comment 7 plaffiazho 2017-11-18 15:09:12 UTC
Jaroslav Rohel, your suggestion also assumes the system is running, whereas the admin could very well be looking into what packages were installed on an inactive installation that exists on another partition, for example.

Comment 8 Basic Six 2017-11-20 11:42:40 UTC
I also opened a bug about dnf logging useless messages (while important messages appear to be missing):
https://bugzilla.redhat.com/show_bug.cgi?id=1364029

That was over a year ago. My bug was closed as a duplicate of bug 1355764, which is just a month older. There are enough complaints and detailed comments. Sadly, nobody seems to be working on this, judging by the lack of serious responses.
Let's hope that some developers will see this bug and improve the software to provide a useful log file similar to the old yum.log.

Comment 9 Jonathan Wakely 2018-04-26 19:05:08 UTC
The dnf history command doesn't show changes made by PackageKit. A single log showing packages installed/updated/erased would be useful.

Comment 10 Fedora End Of Life 2018-05-03 08:29:17 UTC
This message is a reminder that Fedora 26 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 26. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '26'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 26 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

Comment 11 Charles Butterfield 2018-07-08 18:02:58 UTC
Today I attempted to get my new Fedora-28 system working and I have failed to install something critical that I clearly had installed on Fedora-27.  Sadly, although I saved my entire Fedora-27 disk, the log of exactly which packages were installed around the time I solved my problem on Fedora-27 are simply gone, gone, gone, presumably overwritten by the silly rotation.

Given that the DNF devs clearly like (love?) logging, it seems incredible that they have done away with the permanent activity summary log that used to be stored in yum.log.  PLEASE address this ASAP.

Comment 12 Cody 2018-09-15 23:35:17 UTC
(In reply to Jaroslav Rohel from comment #3)
> From my point of view is dnf history command more comfortable for user than
> log file. 

Last I knew installing programs etc. is a task for administrators, not users. You can argue that it's easier but then if that's the case I ask you why there has to be so much logging (the devs of dnf sometimes needing verbose logging isn't a valid reason no matter what anyone says)? And why can't it be simple logging? What use is logging if it's so verbose includng redundant logging (this is something that systemd is also guilty of but I won't even get into that and in any case I configured my system to so that it wouldn't be much of a problem) i.e. it's cluttered with so much information that really isn't necessary? As some others have said /var/log/yum.log was a simple format that showed what was done and specific to the packages only (and yum is a package manager is it not? And dnf is too, right? So why can't you have a simple file like yum did?). It was also very easy to parse and by far easier to parse than the example you give below. As in far, far, far, far easier. 

> 
> You can get list of installed/removed packages from history command too. Try
> to use "dnf history info <ID>" command where <ID> is id of transaction. Eg.
> "dnf history 85"
>

Since when is Unix/Linux meant to only have one way to do something (see above too)? And I ask you this too: if the history subcommand is easier why do you require looking up an ID number - a number! - to pass to the command? That's contradictory. Surely you must realise this?
 
> And now I want more info about transaction 618

I thought the idea was to make it simpler yet here you also show how you have to look up a transaction ID. If you think looking at a very simple log file is difficult for users (though again it's an administration task last I remember) then I can't imagine what this would be like (nor can I figure out how you think this is an improvement or simpler or more comfortable). Actually I can imagine how it would affect users. And it's not the way developers should think; it's a regression and absolutely not the opposite.

Look. I get it. I'm a programmer and I understand all too well how some users are rude/insolent and demand this and that without any consideration for how the programmers feel. But that doesn't mean programmers can't also be guilty of these types of things. The irony is that if both users and programmers realised this and actually followed through it would make things so much easier, less frustrating, more efficient and everything would be much better in general. But it seems that this is impossible. The suggestion that the dnf logging can be compared (truthfully) to yum is just ridiculous at best.

I wouldn't even bother here but as of F28 now yum doesn't work. Isn't that great. I have to ask if it doesn't even work and since it's been deprecated for a long time why is it even there? If you wanted people to migrate to dnf they've had long enough yet you still have it. Why include something that doesn't even work? If I did that with my projects I'd be ashamed of myself and my work. That's just sloppy at best. If you want Linux to be more user friendly you've gone quite the opposite route. 

I (In reply to Jaroslav Rohel from comment #5)
> 
> "for" and "grep" can help.
> Example:
> for i in {515..619} ; do dnf history info $i ; done | grep -E
> 'time|Install|Upgrade|Reinstall|Erase'
> 
> I admit it is not very user-friendly.
> But I think that better solution will be to improve "history" command than
> changing log file.

Why was it pushed out before it was even ready? That's not a good way to develop software (or anything at all for that matter). First you say you think it's easier (and thus better) for users to use the dnf interface and now you're suggesting shell scripting because the dnf interface isn't ready? That's so backwards it's really hard to believe anyone would go that way. It's much worse than 'not very user-friendly'. Why can't you account for everyone? Is it really that hard to have simple logging? That would make things so much better and on my behalf I wouldn't care about the rest. Why can't this be done?

You know what's both funny and sad though? You say that it'd be better to improve the history command than changing the log file. You also suggest bash for loops to deal with it in the meantime. You must realise that if it was simple logging it would be much easier to parse, right? And don't you want it easier? That's the bottom line though: you suggest you want it easier but you've made it quite hte opposite. Why?

Comment 13 Cody 2018-09-16 16:35:43 UTC
...after I wrote the above I realised that part of that might sound like I'm accusing you of being rude. I want to say that that's not what I was trying to say. What I was getting at is that developers and users often don't work together whether polite or impolite or anything else. Everyone here is being cordial but nevertheless it doesn't mean that there is cooperation to account for everyone and that means all users too. The dnf logging problem (and it is a real problem) isn't just here and that should say something. But it seemingly does not matter to the dnf devs - and that's the problem.

Comment 14 Cody 2018-10-10 12:52:30 UTC
I decided to look into this a bit further; I didn't expect to find anything of use but I was pleasantly surprised. So to those who want a simple file it's actually alredy there but for whatever reason the devs don't mention it (or if they do they don't mention it in bug reports like this).

Check the file /var/log/dnf.rpm.log

There unfortunately is one useless log message (for my systemd fixes I was able to prevent it but for whatever reason my rsyslog configuration isn't matching it and I can't be bothered worrying about it when I can simply filter out the one log message). The useless line looks like:

> 2018-10-10T12:22:40Z INFO --- logging initialized ---

But everything else is almost identical, it seems, to /var/log/yum.log with the exception of the timestamp; whereas in /var/log/yum.log you have e.g:

> Jan 04 13:38:18 Installed: kernel-core-4.14.11-200.fc26.x86_64

In dnf.rpm.log it looks like (this isn't the same entry so it's not completely comparable) :

> 2018-03-06T16:20:44Z INFO Installed: kernel-core-4.15.6-300.fc27.x86_64


What's more is it goes back much further (as the above shows). What I don't quite understand is the timestamp; for instance today at 5 something it showed:

> 2018-10-10T12:38:51Z INFO Installed: calc-2.12.6.5-3.fc28.x86_64

But I suspect it's just a matter of looking up (or figuring out) the format of the timestamps themselves. Now it'd only be nice if the devs would point people to this file when they have a request like this in the bug entry...

HTH everyone who's been plagued by this problem (that I and obviously others believe is a regression). This find makes me okay with dnf at least mostly (I still would prefer being able to use `yum' and I still think it extremely silly at best to include `yum' if it doesn't even work - it's sloppy to say the least - but at least there is a file that is very similar to /var/log/yum.log and just as simple to parse).

Cheers.


Kind regards.

Comment 15 Jonathan Wakely 2018-10-10 13:43:56 UTC
As I said in comment 9, updates done via PackageKit still aren't logged anywhere, so I get:

[root@knitphad ~]# wc -l /var/log/dnf.rpm.log
82 /var/log/dnf.rpm.log
[root@knitphad ~]# grep -v 'logging initialized' /var/log/dnf.rpm.log | wc -l
0

(In reply to Cody from comment #14)

> What's more is it goes back much further (as the above shows). What I don't
> quite understand is the timestamp; for instance today at 5 something it
> showed:
> 
> > 2018-10-10T12:38:51Z INFO Installed: calc-2.12.6.5-3.fc28.x86_64
> 
> But I suspect it's just a matter of looking up (or figuring out) the format
> of the timestamps themselves.

That's an ISO 8601 timestamp in UTC.

Comment 16 Cody 2018-10-12 12:30:59 UTC
(In reply to Jonathan Wakely from comment #15)
> As I said in comment 9, updates done via PackageKit still aren't logged
> anywhere, so I get:

Ah - that explains it. I don't use PackageKit (which is probably why I glossed over your comment though looking at it again I recall it quite well); I'm a command line person much more than GUI (though my Fedora box is GUI my CentOS is not and I've lately only been on my laptop - MacBook Pro - as the next sentence might explain I've not really been in my proper place although to be fair I'm in the best place ever in the most important way but there's a temporary - but indeterminate time - indirect issue there). This was hardly important to me even so because in addition to not usig PackageKit; to put things into perspective: the four deaths in the family since 27 January this year aren't the most difficult thing going on atm ...

But as for the logging it's of course easy to filter that out. Had the thought even to make a cronjob that simply filters that out and concatenates to /var/log/yum.log (if any changes relative to the /var/log/yum.log) simply to (1) keep it separate; and (2) to make it as close to before as possible (though there are some differences of course).

> 
> (In reply to Cody from comment #14)
> 
> > What's more is it goes back much further (as the above shows). What I don't
> > quite understand is the timestamp; for instance today at 5 something it
> > showed:
> > 
> > > 2018-10-10T12:38:51Z INFO Installed: calc-2.12.6.5-3.fc28.x86_64
> > 
> > But I suspect it's just a matter of looking up (or figuring out) the format
> > of the timestamps themselves.
> 
> That's an ISO 8601 timestamp in UTC.

Ta. I had the thought of looking into it but it didn't bother me that much (or more correctly it's nothing to some other things going in my life as I said above). I just looked above at this part of your comment and that makes perfect sense now (sleep has also been worse than usual; yesterday woke up before 2:30 and today just after 2:30 but I forgot about the time itself - and yes that's ridiculous but still). It indeed explains why the time was so wrong in my head - it wasn't my time zone!

Appreciate the comment very much! Anyway at least on my behalf this is all good. Well perhaps more like 'good' if you follow me (and I suspect you might very well follow me exactly on that).

Kind regards.

Comment 17 Basic Six 2018-10-22 10:36:52 UTC
(In reply to Cody from comment #14)
> Check the file /var/log/dnf.rpm.log
> 
> There unfortunately is one useless log message (for my systemd fixes I was
> able to prevent it but for whatever reason my rsyslog configuration isn't
> matching it and I can't be bothered worrying about it when I can simply
> filter out the one log message). The useless line looks like:
> 
> > 2018-10-10T12:22:40Z INFO --- logging initialized ---

Again, this entry seems to clutter the log file while other entries seem to be missing.

# grep -cv 'INFO --- logging initialized ---' /var/log/dnf.rpm.log
0

This is shortly after a system upgrade (*upgrade*, not update).

Comment 18 Scott Cohen 2018-11-23 07:27:58 UTC
(In reply to Cody from comment #14)
> What's more is it goes back much further (as the above shows). 

I'm not sure if I'm misinterpreting the context here or not, but regardless, if you have logrotate, this is a problem (if you don't periodically append to yum.log ) as older logs will be deleted every 4 weeks (it seems that is the default period for my install). Does anyone know how regularly the dnf install script overwrites its configuration for logrotate?

Comment 19 Cody 2018-12-06 14:13:26 UTC
(In reply to Basic Six from comment #17)
> Again, this entry seems to clutter the log file while other entries seem to
> be missing.
> 
> # grep -cv 'INFO --- logging initialized ---' /var/log/dnf.rpm.log
> 0
> 
> This is shortly after a system upgrade (*upgrade*, not update).

Yes and a simple filter will be fine won't it? A simple:

# grep -v -- 'INFO --- logging initialized ---' /var/log/dnf.rpm.log

... as you even demonstrated and essentially is what I was saying in other words it's not a problem is it? Want to look at it in a text editor? Redirect it to another file. Write a script to do so if you need or want. Problem solved. This wasn't my point anyway but there you go; at least on my behalf filtering that out and it's more or less equivalent except that the date format is different. Easy enough to change too - write a script or if you prefer a C program to parse it and rewrite the date.

So yes it's cluttered but it's not so much garbage that it's a problem. Maybe it's a problem for you but given you give that example `grep' invocation I don't believe you do have that problem.

Comment 20 Cody 2018-12-06 14:19:58 UTC
(In reply to Scott Cohen from comment #18)
> (In reply to Cody from comment #14)
> > What's more is it goes back much further (as the above shows). 
> 
> I'm not sure if I'm misinterpreting the context here or not, but regardless,
> if you have logrotate, this is a problem (if you don't periodically append
> to yum.log ) as older logs will be deleted every 4 weeks (it seems that is
> the default period for my install). Does anyone know how regularly the dnf
> install script overwrites its configuration for logrotate?

Yes you missed my point in that for me it's not a problem because I specially configure logrotate to not rotate that file except once a year (see below). And iirc it won't change the configuration but write to e.g. a .rpmnew file. Yep, it does though this is on CentOS:

$ ls -1 /etc/logrotate.d/yum.rpmnew 
/etc/logrotate.d/yum.rpmnew

In other words it's not a problem.

As for why I rotate it once a year: because once upon a time I was seriously confused with logwatch showing me installing/updating software in the past 24 hours but I knew very well I hadn't; this of course was concerning so I looked at /var/log/yum.log (again that was on CentOS) and noticed this problem. So I just set the number of logs to keep really high.

So you shouldn't have a problem other than you might want to rotate it once a year.

HTH.

Comment 21 Cody 2018-12-06 14:24:19 UTC
(In reply to Cody from comment #20)
> (In reply to Scott Cohen from comment #18)

> And iirc it won't change the configuration but write to e.g. a
> .rpmnew file. Yep, it does though this is on CentOS:
> 
> $ ls -1 /etc/logrotate.d/yum.rpmnew 
> /etc/logrotate.d/yum.rpmnew

To confirm it's the same with dnf on my Fedora box:

# ls -al /etc/logrotate.d/dnf 
-rw-r--r-- 1 root root 536 29 Dec 09:34:15 2016 /etc/logrotate.d/dnf

Cheers.

Comment 22 Adam Williamson 2019-04-03 17:42:07 UTC
For the record, PackageKit transactions are not logged in dnf.rpm.log because they just, well, don't happen through the path that's logged in dnf.rpm.log.

All three 'dnf' log files - /var/log/dnf.log , /var/log/dnf.librepo.log , and /var/log/dnf.rpm.log - are created and populated by the dnf Python library. So they will only be updated by dnf itself, or other things that use the 'dnf' Python library.

The old PackageKit 'yum' plugin did do quite a lot of stuff through the 'yum' Python library, so probably its install / update / remove transactions wound up being logged in yum.log. But the current PackageKit 'dnf' backend actually uses the libdnf C library, it does not use the 'dnf' Python library. So it bypasses these logging mechanisms.

Any logging that is done at libdnf level should be shared between dnf and PackageKit, but AFAICS libdnf doesn't actually do any default logging to file.

I believe PackageKit transactions *should* end up in the dnf history, at least since the history database updating was moved from dnf itself into libdnf...which was with dnf 3.x, which was Fedora 29. Prior to that, PackageKit transactions weren't in dnf history either.

So, what should be done with this bug? Is there still a demand for an improved log file here?

Comment 24 Jonathan Wakely 2019-04-03 18:01:46 UTC
(In reply to Adam Williamson from comment #22)
> So, what should be done with this bug? Is there still a demand for an
> improved log file here?

I used to use that file to say "what was upgraded since last week, when XYZ was working fine?" and it had everything, whether installed/updated with the dnf cli or by clicking "install updates" when prompted by PackageKit.

I haven't done that for a while, because I can't, so it's obviously possible to live without it. But I used to find it valuable.

Comment 25 Charles Butterfield 2019-04-03 23:09:08 UTC
(In reply to Adam Williamson from comment #22)
> So, what should be done with this bug? Is there still a demand for an
> improved log file here?
I think the title really captures the desires of most posters, namely "produce equivalent to yum.log (log of package installs, updates, and removes".  As to how exactly I see at least two straightforward approaches:
1) Create a separate file (perhaps dnf-yum.log) that focuses solely on tracking changes and which is presumably so small that it never gets rotated (like yum.log)
2) OR assuming all of the relevant information is in the big bloated logfile - add a command that spits out just the yum-like subset (without requiring the user to consult other databases to interpret the results) AND disable rotation of the big bloated logfile to avoid losing the critical tiny subset lurking in the bloat.

I think the first approach makes the most sense, since it would separate the two key use cases: long term permanent change tracking vs short term debugging of DNF itself.

Comment 26 Cody 2019-07-31 23:25:30 UTC
(In reply to Adam Williamson from comment #22)

> So, what should be done with this bug? Is there still a demand for an
> improved log file here?

As the above comment states the title is a pretty good explanation of what should be done. I said that it is more or less 'okay' but it really is okay as far as a workaround rather than it being 'resolved'. It should be simple like it used to be. A simple log file with the packages that were installed, updated, removed etc. is extremely valuable for all sorts of different reasons. I don't understand the confusion here. If you have to have verbose logging then make it configurable (and default should not be very verbose) or have a separate file. But I do not see why there has to be a verbose log file of what packages were installed etc. That seems just silly to me to say the least. I could say more than that but it is more than enough too.

Comment 27 Lukáš Hrázký 2019-08-01 10:53:58 UTC
We are considering switching the logging facility to syslog/journald. In case the decision would be made we want to log the package operations as described here, would it be a good solution to log the package operations also to syslog/journald, possibly under a specific identifier that would allow easy filtering of just these messages?

Comment 28 Cody 2019-08-01 14:22:37 UTC
(In reply to Lukáš Hrázký from comment #27)
> We are considering switching the logging facility to syslog/journald. In
> case the decision would be made we want to log the package operations as
> described here, would it be a good solution to log the package operations
> also to syslog/journald, possibly under a specific identifier that would
> allow easy filtering of just these messages?

On my behalf that seems to be more clutter even without the fact journald is excessive and redundant logging (this includes logging things without context that are logged elsewhere with context). Now maybe that wouldn't become such an issue with package management - I would hope not. That being said depending on how it's done with syslog maybe it would be okay. Over time as I got more and more fed up with system/journald I disabled the journald (as in made it volatile only) and then added additional filters as I saw fit in rsyslog (by way of its RainerScript and possibly the older way). This would mean that depending on what the format of the log message is it would potentially work.

Yet as an admin, user and programmer I have to ask why it cannot be just how it used to be in its own file. It seems to me to be the cleanest way. If however you do migrate it to journald I implore you to please please please do not include all that verbose logging; make it simple like it used to be. That means the following (both from a user as well as a developer i.e. the syslog(3) library function too:

(1) Keep the ident string simple. Maybe "def" itself or otherwise something very simple that can be hooked in rsyslog (and other syslog daemons). See below for an example of what I mean.

(2) Keep the messages brief (example below). 

Example of filter in RainerScript (rsyslog):
--
if $programname == 'dnf' then {
    /var/log/dnf.log
    stop
}
--
Log format example would be just like the old yum.log i.e. in the format of:

--
Jul 31 14:31:13 dnf: Updated: 32:bind-utils-9.9.4-74.el7_6.2.x86_64
Jul 31 14:31:34 dnf: Installed: kernel-3.10.0-957.27.2.el7.x86_64
--

The difference being that the string '<ident>:' is prepended - as syslog(3) function does (specified in the openlog(3) function). But either way none of that verbose irrelevant content. If you must do that then please make it easy to filter out (or configurable) by e.g. regexps in say RainerScript something like:
--
if $programname == 'dnf' then {
    if re_match($msg, "^.*Installed:") then ...
}
--
And I thank you for asking us about this. It gives me a little more hope in Fedora devs. I know that sounds harsh and I do not mean it to be but some of the bugs that seem rather significant are not even addressed and I have had problems with upgrades too. I can say in the example I am thinking of part of that was my fault but because of the grub2 being changed I saw a grub error on boot up and it took me some time to realise what was happening. This is indeed a set of unfortunate circumstances but reacting to bug reports more regularly would be appreciated. So thank you. That is completely sincere. As I noted in a previous comment in this bug report and others as a programmer and user I understand all too well that users can be very difficult to deal with; they provide useless or near useless information, complain and worse of all belittle etc. the devs. The trouble is devs are also susceptible to not being helpful but if both were to work together things would be much better. I consider this bug report to be much more like it should be.

Comment 29 Cody 2019-08-01 14:25:23 UTC
...Obviously I mean 'dnf' where there are references to 'def'. I could have sworn I fixed them all. Not on my Linux box but rather my MacBook Pro...

Comment 30 David Johnston 2020-11-02 02:52:43 UTC
Created attachment 1725653 [details]
A script to pull yum.log out of dnf

This is a first attempt at getting the package histories out of yum.
I wrote it, and I'm releasing it under GPL version 3.

Comment 31 Jaroslav Mracek 2023-08-23 05:48:32 UTC
I guess that DNF cannot provide perfect solution. As a workaround was mentioned `dnf.rpm.log` but it does not contain changes from packagekit, zypper, rpm. I suggest that it is possible to use `dnf history`. It contains all changes performed by dnf, microdnf, and packagekit,  but it does not cover changes performed by RPM or Zypper. The only solution would be to have a simple RPM plugin, that will log all changes on the system.

Comment 32 Jaroslav Mracek 2023-08-23 05:52:28 UTC
I am changing the component, because RPM sounds like a better place to resolve the issue.


From DNf point of view the issue is resolved `dnf.rpm.log` or wontfix because it does not contain information from other package managers.

Comment 33 Panu Matilainen 2023-11-13 08:08:34 UTC
Rpm has two options for logging: rpm-plugin-syslog and rpm-plugin-audit. The latter is far more detailed with things signature checks, which command was used and so on, and accessible with standard audit tools (eg ausearch).

So I don't think we need more logs, but perhaps we should do more (as in, something at all) to ensure people have the audit plugin installed.

Comment 34 Panu Matilainen 2023-11-13 09:20:03 UTC
*** Bug 203796 has been marked as a duplicate of this bug. ***

Comment 35 Panu Matilainen 2023-11-13 09:24:34 UTC
Fixed in rpm-4.19.0-2.fc40: rpm-libs now recommends rpm-plugin-audit so most systems will get it, but with an option to opt out. An alternative would be a hard conditional dependency on audit, but then it becomes a question of what will pull *that* it.

Comment 36 Cody 2023-11-27 12:33:40 UTC
(In reply to Panu Matilainen from comment #35)
> Fixed in rpm-4.19.0-2.fc40: rpm-libs now recommends rpm-plugin-audit so most
> systems will get it, but with an option to opt out. An alternative would be
> a hard conditional dependency on audit, but then it becomes a question of
> what will pull *that* it.

Fixed? This is not fixed. Having to use an alternative program rather than one having to simply parse a text file (like we are all familiar with whether grep, cut, whatever) is not the same thing. Just like binary log files are not nearly as helpful as text. It's not a solution. It's a regression.

And how can something like this at all be equivalent to the yum.log? That was the issue. The fact it doesn't do that means it is NOT fixed. It's just throwing out other (inadequate) alternatives because someone or some people don't want to make the change.

I looked at the syslog plugin and of course it's in /var/log/messages which is not as helpful as in its own log file. The point of logs is to help keep track of things and the way it's set up is not helpful. No way to configure it. Forces more workarounds like with rsyslog. That's not a solution or a fix. That's forcing people who are already busy (and who had a perfectly good answer before) to do more things.

I understand that programming involves risks but I don't understand how this could have happened. It's just not a good answer. The simplicity of the log file of yum is what was so great about it. Why can't you do that for people forced to use dnf, not yum? If I could use yum I would because of this very fact alone but that's not possible now.

Comment 37 Cody 2023-11-27 12:36:50 UTC
(In reply to Panu Matilainen from comment #33)
> Rpm has two options for logging: rpm-plugin-syslog and rpm-plugin-audit. The
> latter is far more detailed with things signature checks, which command was
> used and so on, and accessible with standard audit tools (eg ausearch).
> 
> So I don't think we need more logs, but perhaps we should do more (as in,
> something at all) to ensure people have the audit plugin installed.

.. neither of which are equivalent or even close to the yum.log. So therefore not a solution. The first one is closer but still not same text (fine: that's not necessarily a problem). Problem is it's thrown into /var/log/messages rather than its own file.

The latter requires a special program that comes with it. That's totally not a solution. Why is it so hard to understand that it's not that we need more logs but rather a log file that is SIMPLE to parse and look at like before? The log of dnf has totally irrelevant information that has to be filtered out and frankly is just littering the logs. Like:

2023-11-26T23:57:24-0800 INFO --- logging initialized ---
2023-11-27T00:01:01-0800 INFO --- logging initialized ---
2023-11-27T00:01:14-0800 INFO --- logging initialized ---
2023-11-27T01:22:54-0800 INFO --- logging initialized ---
2023-11-27T02:35:22-0800 INFO --- logging initialized ---
2023-11-27T03:25:12-0800 INFO --- logging initialized ---
2023-11-27T03:59:34-0800 INFO --- logging initialized ---
2023-11-27T03:59:44-0800 INFO --- logging initialized ---

... what's the point of that? That's not helpful and it's annoying. It serves no purpose. Showing that logging has started once is perfectly adequate.

Comment 38 Cody 2023-11-27 12:44:37 UTC
(In reply to Jaroslav Mracek from comment #32)
> I am changing the component, because RPM sounds like a better place to
> resolve the issue.
> 
> 
> From DNf point of view the issue is resolved `dnf.rpm.log` or wontfix
> because it does not contain information from other package managers.

How is the request for a _DNF_ log that matches or similar enough to the old /var/log/yum.log better solved by the RPM package managers? What this sounds like is you just don't want to do it which means it won't happen because those people won't fix a problem with dnf and the problem is with dnf, not rpm. You wouldn't want them to fix it either. If you refuse to fix it why not just say so?

... and if I sound irritated it's because it's so frustrating that we're being told other things that are different are the 'solution', again and again. I I am trying my best to be polite and I am sorry if anything comes across as rude. It's not intentional. I'm just having a hard time unrelated too but I only just saw these things and knew I would not reply if I didn't.

And yes it's 'again and again'. The report was opened over 6 years ago now and such a simple thing (that already existed in the predecessor) should not take that long unless the devs don't want to do it. But if that's the case why not just be upfront about it? Changing it to the rpm devs is just refusing to solve it because as you correctly said the log file is a dnf issue .. and in that same sentence you say that therefore the rpm folks should do it. That boggles my mind.