Bug 584525 - "yum list" output wraps for no particular reason
"yum list" output wraps for no particular reason
Status: CLOSED NOTABUG
Product: Fedora
Classification: Fedora
Component: yum (Show other bugs)
12
All Linux
low Severity medium
: ---
: ---
Assigned To: Seth Vidal
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2010-04-21 15:36 EDT by schnaffman
Modified: 2018-05-09 03:44 EDT (History)
23 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-04-21 16:33:28 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description schnaffman 2010-04-21 15:36:34 EDT
Description of problem:
In a large enough console, "yum list installed" outputs a list of all installed package, one package per line, on 3 columns. On the very same console, "yum list installed > installed.log" outputs the same list with lines no longer than 80 characters, making several entries wrap on a second line.

This behavior can break scripts that would use the output of yum in some way, e.g. "yum list installed | awk '{print $1}'" in order to get a list of package names (without getting the version number or the repo), or scripts that might want to get the version number of some packages like "yum list installed | grep xorg", which will then output xorg-x11-utils.x86_64 with no version attached to it.

Version-Release number of selected component (if applicable):
yum version 3.2.25

How reproducible:
Always

Steps to Reproduce:
1. yum list installed | awk '{print $1}'
  
Actual results:
You get a mix of package names with versions and sometimes repo names. I.E. :

xz-lzma-compat.x86_64
4.999.9-0.1.beta.20091007git.fc12
@anaconda-InstallationRepo-200911081904.x86_64
yelp.x86_64
yp-tools.x86_64
ypbind.x86_64
@anaconda-InstallationRepo-200911081904.x86_64
yum.noarch
yum-metadata-parser.x86_64
1.1.2-14.fc12
yum-presto.noarch
yum-utils.noarch
zd1211-firmware.noarch
1.4-3
zenity.x86_64
zip.x86_64
zlib.x86_64



Expected results:
Only the content of the first column is displayed, namely the package names.

xz-lzma-compat.x86_64
yelp.x86_64
yp-tools.x86_64
ypbind.x86_64
yum.noarch
yum-metadata-parser.x86_64
yum-presto.noarch
yum-utils.noarch
zd1211-firmware.noarch
zenity.x86_64
zip.x86_64
zlib.x86_64
Comment 1 James Antill 2010-04-21 16:33:28 EDT
When you redirect the output, yum cannot find out how big the terminal is ... so it defaults to 80 characters wide (the std.).

I wrote: http://james.fedorapeople.org/yum/plugins/ttysz.py a while ago, which you can use. Or you can change the scripting to use another API (like rpm -qa --qf '%{name}', or even talk directly to yum using python).
Comment 2 Bill Glover 2014-05-01 14:40:44 EDT
A partial work-around for some cases (where you are in an actual terminal).

script -c "yum list installed" > some_file; awk '{print $1}' < some_file

Will work as expected if it's run from a terminal that is wide enough. The colored output is an annoyance, but manageable.

This allowed me to, at least, diff the output of yum from two different systems.
Comment 3 Sander Flobbe 2014-09-28 07:23:04 EDT
This has been reported 'not a bug', but I disagree.

In comparable situations with other tools, e.g. "ps -afe | cat" what you get is an environment with no wrapping limit, which is very suitable for making unix pipelines and smart scripts.

To compare the installed packages on two systems, one should be allowed to pipe the output of 'yum list installed' through tools like sort, grep, awk, diff.

Basic principles of how unix tools can be expected to work, are being violated.
Comment 4 Panu Matilainen 2014-09-29 03:36:43 EDT
Yum is an interactive text-based ui which is not suited, nor intended for piping. Use the right tool for the job - for looking at installed packages rpm queries would do, but the "yum level" tool for scripted queries and piping would be repoquery (from yum-utils).
Comment 5 Paul Jakma 2015-06-20 11:34:55 EDT
I just hit this myself. To close it as notabug because yum is intended as interactive seems petty.

repoquery doesn't print out the same information at all. E.g. it's not obvious how to make it print out the repo-name of installed packages.
Comment 6 buildtherobots 2015-07-08 09:45:19 EDT
It strikes me as worse than petty.

Yum does different things depending on if you pipe/redirect the output or not.

You can't say its not suited or intended for piping as it's actively checking if its output is being piped and only then reacting differently. 
If it's not intended for piping then piping it should _NOT_ cause the output to be different.

rpm is unsuitable as it has no ideas about repos.

repoquery is unsuitable as it requires me to install extra packages. Passively collating a list of _currently_ installed packages should not require changing that list -this isn't quantum physics where the mere act of observation means changing the outcome.

If this really is intended behaviour, then can someone please explain the reasoning behind it? The only reason that seems to exist currently is "because fsck you!"
Comment 7 buildtherobots 2015-07-08 11:43:32 EDT
I've now done some further testing...

>When you redirect the output, yum cannot find out how big the terminal is ... so it defaults to 80 characters wide (the std.).

The above rationale doesn't make sense, isn't consistent and isn't actually true.

Just piping `yum list installed` into less makes it obvious that actually it's happy outputting lines 111 characters long, which is not a standard terminal width.

If I create an RPM with a packageName of 110 characters (and then do a `yum list installed`) yum doesn't split the package name, but instead leaves my terminal to do the wrapping for me. 
So even for columns, it _doesn't_ default to 80 characters, it's more than happy to display 110 on a line and leave the terminal to cope with the output.

It also doesn't do anything to wrap the "From repo" column, so combined with the version column it's once again more than happy to output a single line that's 150 characters in length or 233 characters including the 79 of whitespace.

This seems internally inconsistent and generally just ruddy stupid. As other people have said, this should be a bug; this behaviour seems unjustifiable and wrong.
Comment 8 egetschmann 2015-07-15 14:00:01 EDT
Hi Redhat, 

I'm looking to use the contents of yum check-update in a script.  Getting around this BUG is going to be a real pain. I don't see any obvious way to get this information from repoquery.  Are there other alternatives you are suggesting, to be able to sanely check whether there are currently updates available on a repo?
Comment 9 egetschmann 2015-07-15 14:18:43 EDT
A good work around listed here  - http://fedora.12.x6.nabble.com/yum-list-display-some-package-in-2-rows-td2496921.html

work around is to pipe your yum output through this to remove the newlines that have whitespace after them.  Thanks to that poster, Allen Kistler! 

tr "\n" "#" | sed -e 's/# / /g' | tr "#" "\n"
Comment 10 John 2017-03-11 12:34:54 EST
Yeah that tr & sed hack works, although it's still a bit of a dog, as the joined lines now have a load of extra padding space that was added onto the wrapped line by yum/dnf, so they're wider than they should be. At least they're joined onto one line though, so the output can be sorted properly.

For convenience u can add this function to your bashrc, and then use dnfunwrap to call any dnf commands you want with the output unwrapped:

function dnfunwrap {
    dnf $@ | tr "\n" "#" | sed -e 's/# / /g' | tr "#" "\n"
}


For developers to give this cop-out that yum/dnf are interactive tools is utter nonsense.

Also note: 
 "Panu Matilainen 2014-09-29 03:36:43 EDT
Yum is an interactive text-based ui which is not suited, nor intended for piping. Use the right tool for the job - for looking at installed packages rpm queries would do, but the "yum level" tool for scripted queries and piping would be repoquery (from yum-utils)."

Well, guess what - repoquery/yum-utils are now deprecated & redirect to dnf with the reqoquery subcommand.

With dnf absorbing reqpoquery, and its myriad of other options, dnf is unquestionably intended to be used by in "non-interactive" scripts.

All deployment tools should be moving towards modes that are scriptable and automatable.

Just, for god's sake, add an option to dnf to disable this absurd wrapping. It makes it so painful to sort & grep the output of dnf list commands.
Comment 11 FroL_Onn 2017-04-13 03:32:41 EDT
That is so inconvenient!
This wrapping occurs for no sensible reason and scrambles output making it less usable. I don't understand how this could be "NOTABUG".

If it is a feature, what is a reason for the feature to be introduced?
And if the reason is really strong should an appropriate feature requests be sent for ls, ps etc...?

Finally, if it is a feature, could a simple flag be created to turn it on and off?
Comment 12 Harald Gutmann 2017-06-06 02:21:37 EDT
I also just stumbled into this problem, and have to state that this is *not* the way I would expect a list to be wrapped when beeing piped into another program.


In fact it makes any scripting with dnf outputs ways more prone to mistakes and errors. I really have a hard time understanding that this is the desired default behaviour.

In my point of view it should be considered to add a command line switch / configuration item to change that behaviour and even to make that option the default operation mode.

Kindly consider reopening this issue.


Best regards,
Harald Gutmann
Comment 13 Brian J. Murrell 2017-06-08 20:43:26 EDT
Have to pile on here and say simply that piped output should not consider terminal width at all.

If you want to detect that output is going to the terminal and format for it's dimensions accordingly, that is fine.

But when output is being piped/redirected somewhere else, all prettyifying needs to be avoided!
Comment 14 Brian J. Murrell 2017-06-08 21:20:05 EDT
A bit more research has yielded:

$ repoquery -q -a --qf="%{name} %{version}-%{release}.%{arch} %{repoid}" --pkgnarrow=updates

for yum check-update.

Maybe this was the intended reason for closing this NOTABUG.  The above would have been a good response to comment #8 rather than leaving that dangling.
Comment 15 tal 2017-06-28 11:07:45 EDT
Yes, comment #14 is a lot better, however:

1. It is ignorant of yum-lock.
2. It doesn't handle excludes the same way.
3. It doesn't handle repository excludes.

So, basically I'll repeat the feature request (not a bug, a feature request) for "yum check-update -q" to have a flag that outputs one line per package no matter what.  (You can think of this as "assumes an infinitely wide terminal")
Comment 16 George Marselis 2017-07-10 17:34:17 EDT
I will add my 2 bits here and say that this does not go along with the principle of least surprise.

This bug has been haunting me ever since 2012, when I tried to parse the output for a nagios script I was writing. yum/dnf are definitely not scripting friendly

I would second comment #15 that such a feature be implemented and left to the user to configure the behavior.
Comment 17 tal 2017-07-10 18:08:53 EDT
Would the yum maintainers be open to a PR that fixes this problem? I will volunteer.
Comment 18 BugMasta 2017-07-13 02:33:58 EDT
Look at the comment in 2010 from James Antill, who originally closed this bug, on the same day the bug was opened.

James wrote a yum plugin which looks like it resolves the issue perfectly. It's one file. 47 lines, minus the comments.

But 7 years later, this solution has been forgotten and will no longer work with dnf.

The solution needs to be updated and integrated into DNF, so it doesn't get left behind.
Comment 19 BugMasta 2017-07-13 05:56:18 EDT
Having had a look at the documentation for dnf plugins, no surprise, it's an absolute disgrace.

What is the point of having a plugin API if it's not documented well enough for anyone to use.

Utterly hopeless.
Comment 20 John 2017-07-17 09:17:47 EDT
I've done a fix for this, inspired by James Antill's yum plugin:

You can get the new dnf plugin here:

https://github.com/SupremeGit/public/blob/master/red_hat_bugzilla/rh-bz-584525/ttywidth.py
Comment 21 George Marselis 2017-07-29 15:40:02 EDT
John: Thank you!

Does it remove the column width by default?

How can I install that?

Also, how can this be rolled into next official dnf release?
Comment 22 John Marshall 2017-08-01 04:43:52 EDT
We too ran into this yesterday. We are not Python programmers or yum or dnf experts, so for us figuring out how to wield plugins is less attractive than other workarounds like the tr|sed|tr one mentioned above.

IMHO the formatting defaults of other tools (e.g. dpkg) are more common and more useful. If a command is interactive (stdout is a terminal), sure, use fancy formatting aimed at a $COLUMNS-wide terminal.  If stdout is being redirected to a file or pipe, then guess what maybe a script will be reading this, so use simple *predictable* formatting.
Comment 23 BugMasta 2017-08-01 06:03:20 EDT
(In reply to George Marselis from comment #21)
> John: Thank you!

no problemo, glad to be of service :-)

> Does it remove the column width by default?

It doesn't remove the terminal-width formatting, but you can set the terminal-width to be as wide as you like, so that does the trick.

From looking at the dnf python code, the logic to format the output is quite intricate, with formatting of colours, dividing the terminal width into multiple columns, and so on, and it's difficult to bypass all of it. But very easy to just set the terminal width so wide that at least everything will fit on one line, which is sufficient for it to be easily parsed by scripts.
 
> How can I install that?

Copy it into the dnf-plugins directory under your python installation. Eg for Fedora 26, it is:
/usr/lib/python3.6/site-packages/dnf-plugins/ttywidth.py

Other distros will likely use different python version so the location will be a little different.

> Also, how can this be rolled into next official dnf release?

Yep that would be nice.
Comment 24 BugMasta 2017-08-01 06:05:15 EDT
(In reply to John Marshall from comment #22)
> We too ran into this yesterday. We are not Python programmers or yum or dnf
> experts, so for us figuring out how to wield plugins is less attractive than
> other workarounds like the tr|sed|tr one mentioned above.
> 
> IMHO the formatting defaults of other tools (e.g. dpkg) are more common and
> more useful. If a command is interactive (stdout is a terminal), sure, use
> fancy formatting aimed at a $COLUMNS-wide terminal.  If stdout is being
> redirected to a file or pipe, then guess what maybe a script will be reading
> this, so use simple *predictable* formatting.

Totally agree.
Comment 25 Panu Matilainen 2017-08-01 07:34:56 EDT
Discussing dnf issues in a years old yum bug is going to miss the attention of many relevant people, open a new bug on dnf instead.

Oh and FWIW, I agree assuming anything about columns when not on a tty seems wrong.
Comment 26 George Marselis 2017-08-09 10:13:00 EDT
Comment 23

BugMasta: I installed the plugin.

this is what I see:

http://imgur.com/8bWaiL1

The above picture wraps the '='s around to 300 chars. Which leaves me to believe that programmer wanted to ensure that the output is pretty/consistent *AFTER* the pipe.

The correct way is to read the term size before piping, format the output for *THAT* size and then pipe the formatted text to the pipe.

This is, of course, a huge BRILLANT BEAN idea.

Debian. I'm just sayin'
Comment 27 BugMasta 2017-08-09 11:30:21 EDT
(In reply to George Marselis from comment #26)
> Comment 23
> 
> BugMasta: I installed the plugin.
> 
> this is what I see:
> 
> http://imgur.com/8bWaiL1
> 
> The above picture wraps the '='s around to 300 chars. Which leaves me to
> believe that programmer wanted to ensure that the output is
> pretty/consistent *AFTER* the pipe.
> 
> The correct way is to read the term size before piping, format the output
> for *THAT* size and then pipe the formatted text to the pipe.
> 
> This is, of course, a huge BRILLANT BEAN idea.
> 
> Debian. I'm just sayin'

Sorry, I don't have the faintest idea what you're talking about?
Comment 28 starsareblueandfaraway 2017-12-11 19:04:21 EST
dnf repoquery --installed
Comment 29 John 2017-12-11 21:56:34 EST
AHA!

As mentioned by brian in comment 14 you can use repoqueryBUT, as tal mentioned in comment 14:

------------------------------------------
"tal@whatexit.org 2017-06-28 11:07:45 EDT
Yes, comment #14 is a lot better, however:

1. It is ignorant of yum-lock.
2. It doesn't handle excludes the same way.
3. It doesn't handle repository excludes.

So, basically I'll repeat the feature request (not a bug, a feature request) for "yum check-update -q" to have a flag that outputs one line per package no matter what.  (You can think of this as "assumes an infinitely wide terminal")"
-----------------------------------------


But that was when reqpoquery was a separate, apparently deprecated tool.

Now repoquery is IN DNF!! I had no idea. When did that happen? Now it's part of DNF it should avoid some of the issues tal mentioned.

----------
dnf repoquery --queryformat '%{name}.%{arch} : %{reponame}' enlightenment
Last metadata expiration check: 0:12:14 ago on Tue 12 Dec 2017 12:54:52 ACDT.
enlightenment.i686 : fedora
enlightenment.x86_64 : fedora

Or: 

[root@Il-Duce 12-12 13:07:07 qemu]# dnf repoquery --queryformat "%{name} %{version}-%{release}.%{arch} %{repoid}" enlightenment
Last metadata expiration check: 0:12:22 ago on Tue 12 Dec 2017 12:54:52 ACDT.
enlightenment 0.21.8-1.fc26.i686 fedora
enlightenment 0.21.8-1.fc26.x86_64 fedora
---------

I mean, that is great, it's giving info one line per package, with the repository, versiokn numbers etc.


The only problem with this, is that it is querying what's available. It is NOT returning the results of a REQUEST TO INSTALL SOMETHING

Sometimes you just want to use DNF to install something, and you want to *FOR EXAMPLE* grep the results to see whether packages are being pulled in from an undesired repo. So you run 

dnf install balls | grep -i badrepo

But you may get kicked in the balls, because if badrepo contains packages with very long names, the output will wrap, and the grep will drop the package names on the first line, and you just get a dozen lines saying "badrepo" WITHOUT being able to see what package deps were going to be pulled from that repo. 

WE NEED TO BE ABLE TO GREP THE OUTPUT OF "DNF INSTALL", WITHOUT BEING SCREWED BY LINE-WRAPPING.

Until we can do that, this problem with dnf is STILL A MASSIVE PAIN IN THE ASS (and the plugin to widen the terminal is still the only solution). 

And I'm still not comvinced the plugin i did is completely bug-free. It is a tiny plugin though. It should be an official part of DNF and maintained so it doesn't break in future.
Comment 30 Pavel Roskin 2017-12-13 14:45:46 EST
I don't see why dnf should assume the line length to be 80 characters when the output is not a terminal. dnf can trivially check that the output is not a terminal, in particular not a dumb terminal.

There are plenty of tools that are not designed for non-interactive use, yet they don't make it harder to use them in a pipe.

dnf is intentionally applying wrapping to the output that doesn't need it. That is a bug.
Comment 31 John 2017-12-13 23:04:24 EST
+1 to Pavel. It's not rocket science.

"dnf install" is the command we use to install software.

Therefore, we need to be able to grep the output of "dnf install". 

Not "dnf repoquery", or "dnf search", or any other command. Those commands do not perform the function or produce the output that "dnf install" does.

I'll say it one last time:

*WE NEED TO BE ABLE TO GREP THE OUTPUT OF "DNF INSTALL".

Which means, "dnf install *MUST* have an option, or plugin, to output one line per package, with NO wrapping of packages onto a 2nd line. End of story.

Anyone who cannot understand this, should not be developing software for Unix/Linux.
Comment 32 George Marselis 2017-12-14 07:45:34 EST
How can we get the dev to reopen this ticket?
Comment 33 Erik Logtenberg 2017-12-21 18:06:03 EST
Please reopen and fix, it would be greatly appreciated by many people. Thanks.
Comment 34 Dragan 2018-01-14 22:17:56 EST
Particularly nasty behaviour for a common task with dnf:
dnf search golang | less 

How is this a NOTABUG is beyond common sense.
Comment 35 George Marselis 2018-04-29 05:46:43 EDT
/^\w+[\w\.\-]+\.(x86_64|noarch|i[356]86){1}\s+\d[:\.\d]+\.el6(_\d?\.\d)?\s+\w+[-\w]*\w$/

Who in their right mind thinks it perfectly fine to use the above regex in order to grab a single freaking list of the installed packages...

I got 99 problems and yum is one.
Comment 36 John 2018-04-30 08:16:13 EDT
(In reply to Erik Logtenberg from comment #33)
> Please reopen and fix, it would be greatly appreciated by many people.
> Thanks.

If you have a look at the code for dnf, you'll see why the dev, Seth Vidal, is ignoring the bug report. The code is an absolute dog's breakfastt.

You *would* think that just turning off wrapping when the output is not to a terminal, would be an easy thing. But... the code is such a mess, and the code to format the output is smeared across and throughout so much of the codebase, that it's way more difficult than it should be, to just turn off wrapping.
Comment 37 johnm 2018-05-09 03:26:36 EDT
Back to the original issue: we just raised a case with Red Hat against RHEL5 ELS, because yum on two hosts (out of hundreds) was introducing linebreaks if piped or redirected.  
(Minimal) RCA was that these two, alone, had the downloadonly plugin enabled.  Disabling that desensitised yum to pipes and redirects.

cat /etc/yum/pluginconf.d/downloadonly.conf
[main]
enabled=0

Perhaps that will help someone else...
Comment 38 adm.fkt.physik 2018-05-09 03:44:06 EDT
Using the unbuffer command from the expect package is a workaround which works for us:

unbuffer yum list ...

Note You need to log in before you can comment on or make changes to this bug.