Bug 584525

When you redirect the output, yum cannot find out how big the terminal is ... so it defaults to 80 characters wide (the std.).

I wrote: http://james.fedorapeople.org/yum/plugins/ttysz.py a while ago, which you can use. Or you can change the scripting to use another API (like rpm -qa --qf '%{name}', or even talk directly to yum using python).

A partial work-around for some cases (where you are in an actual terminal).

script -c "yum list installed" > some_file; awk '{print $1}' < some_file

Will work as expected if it's run from a terminal that is wide enough. The colored output is an annoyance, but manageable.

This allowed me to, at least, diff the output of yum from two different systems.

This has been reported 'not a bug', but I disagree.

In comparable situations with other tools, e.g. "ps -afe | cat" what you get is an environment with no wrapping limit, which is very suitable for making unix pipelines and smart scripts.

To compare the installed packages on two systems, one should be allowed to pipe the output of 'yum list installed' through tools like sort, grep, awk, diff.

Basic principles of how unix tools can be expected to work, are being violated.

Yum is an interactive text-based ui which is not suited, nor intended for piping. Use the right tool for the job - for looking at installed packages rpm queries would do, but the "yum level" tool for scripted queries and piping would be repoquery (from yum-utils).

I just hit this myself. To close it as notabug because yum is intended as interactive seems petty.

repoquery doesn't print out the same information at all. E.g. it's not obvious how to make it print out the repo-name of installed packages.

It strikes me as worse than petty.

Yum does different things depending on if you pipe/redirect the output or not.

You can't say its not suited or intended for piping as it's actively checking if its output is being piped and only then reacting differently. 
If it's not intended for piping then piping it should _NOT_ cause the output to be different.

rpm is unsuitable as it has no ideas about repos.

repoquery is unsuitable as it requires me to install extra packages. Passively collating a list of _currently_ installed packages should not require changing that list -this isn't quantum physics where the mere act of observation means changing the outcome.

If this really is intended behaviour, then can someone please explain the reasoning behind it? The only reason that seems to exist currently is "because fsck you!"

I've now done some further testing...

>When you redirect the output, yum cannot find out how big the terminal is ... so it defaults to 80 characters wide (the std.).

The above rationale doesn't make sense, isn't consistent and isn't actually true.

Just piping `yum list installed` into less makes it obvious that actually it's happy outputting lines 111 characters long, which is not a standard terminal width.

If I create an RPM with a packageName of 110 characters (and then do a `yum list installed`) yum doesn't split the package name, but instead leaves my terminal to do the wrapping for me. 
So even for columns, it _doesn't_ default to 80 characters, it's more than happy to display 110 on a line and leave the terminal to cope with the output.

It also doesn't do anything to wrap the "From repo" column, so combined with the version column it's once again more than happy to output a single line that's 150 characters in length or 233 characters including the 79 of whitespace.

This seems internally inconsistent and generally just ruddy stupid. As other people have said, this should be a bug; this behaviour seems unjustifiable and wrong.

Hi Redhat, 

I'm looking to use the contents of yum check-update in a script.  Getting around this BUG is going to be a real pain. I don't see any obvious way to get this information from repoquery.  Are there other alternatives you are suggesting, to be able to sanely check whether there are currently updates available on a repo?

A good work around listed here  - http://fedora.12.x6.nabble.com/yum-list-display-some-package-in-2-rows-td2496921.html

work around is to pipe your yum output through this to remove the newlines that have whitespace after them.  Thanks to that poster, Allen Kistler! 

tr "\n" "#" | sed -e 's/# / /g' | tr "#" "\n"

Yeah that tr & sed hack works, although it's still a bit of a dog, as the joined lines now have a load of extra padding space that was added onto the wrapped line by yum/dnf, so they're wider than they should be. At least they're joined onto one line though, so the output can be sorted properly.

For convenience u can add this function to your bashrc, and then use dnfunwrap to call any dnf commands you want with the output unwrapped:

function dnfunwrap {
    dnf $@ | tr "\n" "#" | sed -e 's/# / /g' | tr "#" "\n"
}


For developers to give this cop-out that yum/dnf are interactive tools is utter nonsense.

Also note: 
 "Panu Matilainen 2014-09-29 03:36:43 EDT
Yum is an interactive text-based ui which is not suited, nor intended for piping. Use the right tool for the job - for looking at installed packages rpm queries would do, but the "yum level" tool for scripted queries and piping would be repoquery (from yum-utils)."

Well, guess what - repoquery/yum-utils are now deprecated & redirect to dnf with the reqoquery subcommand.

With dnf absorbing reqpoquery, and its myriad of other options, dnf is unquestionably intended to be used by in "non-interactive" scripts.

All deployment tools should be moving towards modes that are scriptable and automatable.

Just, for god's sake, add an option to dnf to disable this absurd wrapping. It makes it so painful to sort & grep the output of dnf list commands.

That is so inconvenient!
This wrapping occurs for no sensible reason and scrambles output making it less usable. I don't understand how this could be "NOTABUG".

If it is a feature, what is a reason for the feature to be introduced?
And if the reason is really strong should an appropriate feature requests be sent for ls, ps etc...?

Finally, if it is a feature, could a simple flag be created to turn it on and off?

I also just stumbled into this problem, and have to state that this is *not* the way I would expect a list to be wrapped when beeing piped into another program.


In fact it makes any scripting with dnf outputs ways more prone to mistakes and errors. I really have a hard time understanding that this is the desired default behaviour.

In my point of view it should be considered to add a command line switch / configuration item to change that behaviour and even to make that option the default operation mode.

Kindly consider reopening this issue.


Best regards,
Harald Gutmann

Have to pile on here and say simply that piped output should not consider terminal width at all.

If you want to detect that output is going to the terminal and format for it's dimensions accordingly, that is fine.

But when output is being piped/redirected somewhere else, all prettyifying needs to be avoided!

A bit more research has yielded:

$ repoquery -q -a --qf="%{name} %{version}-%{release}.%{arch} %{repoid}" --pkgnarrow=updates

for yum check-update.

Maybe this was the intended reason for closing this NOTABUG.  The above would have been a good response to comment #8 rather than leaving that dangling.

Yes, comment #14 is a lot better, however:

1. It is ignorant of yum-lock.
2. It doesn't handle excludes the same way.
3. It doesn't handle repository excludes.

So, basically I'll repeat the feature request (not a bug, a feature request) for "yum check-update -q" to have a flag that outputs one line per package no matter what.  (You can think of this as "assumes an infinitely wide terminal")

I will add my 2 bits here and say that this does not go along with the principle of least surprise.

This bug has been haunting me ever since 2012, when I tried to parse the output for a nagios script I was writing. yum/dnf are definitely not scripting friendly

I would second comment #15 that such a feature be implemented and left to the user to configure the behavior.

Would the yum maintainers be open to a PR that fixes this problem? I will volunteer.

Look at the comment in 2010 from James Antill, who originally closed this bug, on the same day the bug was opened.

James wrote a yum plugin which looks like it resolves the issue perfectly. It's one file. 47 lines, minus the comments.

But 7 years later, this solution has been forgotten and will no longer work with dnf.

The solution needs to be updated and integrated into DNF, so it doesn't get left behind.

Having had a look at the documentation for dnf plugins, no surprise, it's an absolute disgrace.

What is the point of having a plugin API if it's not documented well enough for anyone to use.

Utterly hopeless.

I've done a fix for this, inspired by James Antill's yum plugin:

You can get the new dnf plugin here:

https://github.com/SupremeGit/public/blob/master/red_hat_bugzilla/rh-bz-584525/ttywidth.py

John: Thank you!

Does it remove the column width by default?

How can I install that?

Also, how can this be rolled into next official dnf release?

We too ran into this yesterday. We are not Python programmers or yum or dnf experts, so for us figuring out how to wield plugins is less attractive than other workarounds like the tr|sed|tr one mentioned above.

IMHO the formatting defaults of other tools (e.g. dpkg) are more common and more useful. If a command is interactive (stdout is a terminal), sure, use fancy formatting aimed at a $COLUMNS-wide terminal.  If stdout is being redirected to a file or pipe, then guess what maybe a script will be reading this, so use simple *predictable* formatting.

(In reply to George Marselis from comment #21)
> John: Thank you!

no problemo, glad to be of service :-)

> Does it remove the column width by default?

It doesn't remove the terminal-width formatting, but you can set the terminal-width to be as wide as you like, so that does the trick.

From looking at the dnf python code, the logic to format the output is quite intricate, with formatting of colours, dividing the terminal width into multiple columns, and so on, and it's difficult to bypass all of it. But very easy to just set the terminal width so wide that at least everything will fit on one line, which is sufficient for it to be easily parsed by scripts.
 
> How can I install that?

Copy it into the dnf-plugins directory under your python installation. Eg for Fedora 26, it is:
/usr/lib/python3.6/site-packages/dnf-plugins/ttywidth.py

Other distros will likely use different python version so the location will be a little different.

> Also, how can this be rolled into next official dnf release?

Yep that would be nice.

(In reply to John Marshall from comment #22)
> We too ran into this yesterday. We are not Python programmers or yum or dnf
> experts, so for us figuring out how to wield plugins is less attractive than
> other workarounds like the tr|sed|tr one mentioned above.
> 
> IMHO the formatting defaults of other tools (e.g. dpkg) are more common and
> more useful. If a command is interactive (stdout is a terminal), sure, use
> fancy formatting aimed at a $COLUMNS-wide terminal.  If stdout is being
> redirected to a file or pipe, then guess what maybe a script will be reading
> this, so use simple *predictable* formatting.

Totally agree.

Discussing dnf issues in a years old yum bug is going to miss the attention of many relevant people, open a new bug on dnf instead.

Oh and FWIW, I agree assuming anything about columns when not on a tty seems wrong.

Comment 23

BugMasta: I installed the plugin.

this is what I see:

http://imgur.com/8bWaiL1

The above picture wraps the '='s around to 300 chars. Which leaves me to believe that programmer wanted to ensure that the output is pretty/consistent *AFTER* the pipe.

The correct way is to read the term size before piping, format the output for *THAT* size and then pipe the formatted text to the pipe.

This is, of course, a huge BRILLANT BEAN idea.

Debian. I'm just sayin'

(In reply to George Marselis from comment #26)
> Comment 23
> 
> BugMasta: I installed the plugin.
> 
> this is what I see:
> 
> http://imgur.com/8bWaiL1
> 
> The above picture wraps the '='s around to 300 chars. Which leaves me to
> believe that programmer wanted to ensure that the output is
> pretty/consistent *AFTER* the pipe.
> 
> The correct way is to read the term size before piping, format the output
> for *THAT* size and then pipe the formatted text to the pipe.
> 
> This is, of course, a huge BRILLANT BEAN idea.
> 
> Debian. I'm just sayin'

Sorry, I don't have the faintest idea what you're talking about?

dnf repoquery --installed

AHA!

As mentioned by brian in comment 14 you can use repoqueryBUT, as tal mentioned in comment 14:

------------------------------------------
"tal 2017-06-28 11:07:45 EDT
Yes, comment #14 is a lot better, however:

1. It is ignorant of yum-lock.
2. It doesn't handle excludes the same way.
3. It doesn't handle repository excludes.

So, basically I'll repeat the feature request (not a bug, a feature request) for "yum check-update -q" to have a flag that outputs one line per package no matter what.  (You can think of this as "assumes an infinitely wide terminal")"
-----------------------------------------


But that was when reqpoquery was a separate, apparently deprecated tool.

Now repoquery is IN DNF!! I had no idea. When did that happen? Now it's part of DNF it should avoid some of the issues tal mentioned.

----------
dnf repoquery --queryformat '%{name}.%{arch} : %{reponame}' enlightenment
Last metadata expiration check: 0:12:14 ago on Tue 12 Dec 2017 12:54:52 ACDT.
enlightenment.i686 : fedora
enlightenment.x86_64 : fedora

Or: 

[root@Il-Duce 12-12 13:07:07 qemu]# dnf repoquery --queryformat "%{name} %{version}-%{release}.%{arch} %{repoid}" enlightenment
Last metadata expiration check: 0:12:22 ago on Tue 12 Dec 2017 12:54:52 ACDT.
enlightenment 0.21.8-1.fc26.i686 fedora
enlightenment 0.21.8-1.fc26.x86_64 fedora
---------

I mean, that is great, it's giving info one line per package, with the repository, versiokn numbers etc.


The only problem with this, is that it is querying what's available. It is NOT returning the results of a REQUEST TO INSTALL SOMETHING

Sometimes you just want to use DNF to install something, and you want to *FOR EXAMPLE* grep the results to see whether packages are being pulled in from an undesired repo. So you run 

dnf install balls | grep -i badrepo

But you may get kicked in the balls, because if badrepo contains packages with very long names, the output will wrap, and the grep will drop the package names on the first line, and you just get a dozen lines saying "badrepo" WITHOUT being able to see what package deps were going to be pulled from that repo. 

WE NEED TO BE ABLE TO GREP THE OUTPUT OF "DNF INSTALL", WITHOUT BEING SCREWED BY LINE-WRAPPING.

Until we can do that, this problem with dnf is STILL A MASSIVE PAIN IN THE ASS (and the plugin to widen the terminal is still the only solution). 

And I'm still not comvinced the plugin i did is completely bug-free. It is a tiny plugin though. It should be an official part of DNF and maintained so it doesn't break in future.

I don't see why dnf should assume the line length to be 80 characters when the output is not a terminal. dnf can trivially check that the output is not a terminal, in particular not a dumb terminal.

There are plenty of tools that are not designed for non-interactive use, yet they don't make it harder to use them in a pipe.

dnf is intentionally applying wrapping to the output that doesn't need it. That is a bug.

+1 to Pavel. It's not rocket science.

"dnf install" is the command we use to install software.

Therefore, we need to be able to grep the output of "dnf install". 

Not "dnf repoquery", or "dnf search", or any other command. Those commands do not perform the function or produce the output that "dnf install" does.

I'll say it one last time:

*WE NEED TO BE ABLE TO GREP THE OUTPUT OF "DNF INSTALL".

Which means, "dnf install *MUST* have an option, or plugin, to output one line per package, with NO wrapping of packages onto a 2nd line. End of story.

Anyone who cannot understand this, should not be developing software for Unix/Linux.

How can we get the dev to reopen this ticket?

Please reopen and fix, it would be greatly appreciated by many people. Thanks.

Particularly nasty behaviour for a common task with dnf:
dnf search golang | less 

How is this a NOTABUG is beyond common sense.

/^\w+[\w\.\-]+\.(x86_64|noarch|i[356]86){1}\s+\d[:\.\d]+\.el6(_\d?\.\d)?\s+\w+[-\w]*\w$/

Who in their right mind thinks it perfectly fine to use the above regex in order to grab a single freaking list of the installed packages...

I got 99 problems and yum is one.

(In reply to Erik Logtenberg from comment #33)
> Please reopen and fix, it would be greatly appreciated by many people.
> Thanks.

If you have a look at the code for dnf, you'll see why the dev, Seth Vidal, is ignoring the bug report. The code is an absolute dog's breakfastt.

You *would* think that just turning off wrapping when the output is not to a terminal, would be an easy thing. But... the code is such a mess, and the code to format the output is smeared across and throughout so much of the codebase, that it's way more difficult than it should be, to just turn off wrapping.

Back to the original issue: we just raised a case with Red Hat against RHEL5 ELS, because yum on two hosts (out of hundreds) was introducing linebreaks if piped or redirected.  
(Minimal) RCA was that these two, alone, had the downloadonly plugin enabled.  Disabling that desensitised yum to pipes and redirects.

cat /etc/yum/pluginconf.d/downloadonly.conf
[main]
enabled=0

Perhaps that will help someone else...

Using the unbuffer command from the expect package is a workaround which works for us:

unbuffer yum list ...

This is how some users fixes this issue on their own: https://unix.stackexchange.com/questions/308731/how-to-remove-line-wrapping-from-dnf-yum-commands

Yeah, "dnf list installed | sed -r -e :a -e '$!N;s/\n[[:space:]]+/ /;ta' -e 'P;D' | grep @rpmfusion-free", very user-friendly... ;-(

So I second that there should be an option for "straight-ahead" formatting which could be easily processed with other tools.

It seems there's some traction on this issue, but on this bug report:
https://bugzilla.redhat.com/show_bug.cgi?id=1577889&GoAheadAndLogIn=1

Basic principles of how unix tools can be expected to work, are being violated.  +1

For developers to give this cop-out that yum/dnf are interactive tools is utter nonsense.  +1

This does not go along with the principle of least surprise  +1

Using the unbuffer command from the expect package is a workaround which works for us:  unbuffer yum list ...  +1

This behaviour is just crazy and utterly unhelpful. Who is limited to 80 column terminals nowadays.  Even if they are, yum does not actually limit the output to 80 columns anyway, as long repo names extend beyond column 80.

Please at least add an option to keep each row of data on a single line – something like --singlelineformat

The stupidity in yum has now reached a new peak.

I'm doing yum check-update, and, after update to yum-3.4.3-168.el7.noarch, when the output is redirected to a file yum is now wrapping each rpm onto 3 lines.

THREE LINES.

ie, i am now seeing this, in file:

bind-export-libs.x86_64
                   32:9.11.4-26.P2.el7_9.13
                                      susemanager:suma4-redhat-7-2023-03-02-base
bind-libs.x86_64   32:9.11.4-26.P2.el7_9.13
                                      susemanager:suma4-redhat-7-2023-03-02-base

and so on.

The stupidity in Red Hat products just never ceases to amaze me.

@john.sincock.com Why not just switch to DNF?

(In reply to Brian J. Murrell from comment #43)
> @john.sincock.com Why not just switch to DNF?

Because red hat enterprise linux 7 uses yum, not dnf.
Just because dnf might be available at dodgyrepo.com or even in epel doesn't mean it is advisable to use it.

Although, it is a tempting idea, now you mention it... especially for simple things like checking what updates are available...

(In reply to John from comment #44)
> 
> Because red hat enterprise linux 7 uses yum, not dnf.
> Just because dnf might be available at dodgyrepo.com or even in epel doesn't
> mean it is advisable to use it.

I believe it's in 'Extras', so the comparison to dodyrepo.com I don't think is fair.

> Although, it is a tempting idea, now you mention it... especially for simple
> things like checking what updates are available...

We use it for everything including the myriad of package checking/installation/removals/upgrades/etc. that we do in our CI pipeline.  It has not let us down.