Bug 1573676 - ncurses 6.1 breaks mouse in (older?) terminals
Summary: ncurses 6.1 breaks mouse in (older?) terminals
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora
Classification: Fedora
Component: vte
Version: 28
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Kevin Fenzi
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-05-01 22:42 UTC by Leszek Matok
Modified: 2019-05-28 22:59 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-05-28 22:59:25 UTC
Type: Bug


Attachments (Terms of Use)

Description Leszek Matok 2018-05-01 22:42:25 UTC
Description of problem:
With ncurses 6.1, mouse-aware console programs like mc or htop don't recognize mouse clicks (and there are leftover characters making the programs misbehave).

I've installed xterm to test and that indeed works, but found out all vte-based terminals (lilyterm, termit, lxterminal...) produce strange results.

Version-Release number of selected component (if applicable):
ncurses-6.1-4.20180224.fc28.x86_64
vte-0.28.2-23.fc28.x86_64


Work-around is to downgrade to ncurses-6.0-13.20170722.fc27.x86_64 or use swap other terminfo entry for xterm (but which?) I've tried xterm-x11mouse (is xterm's mouse not x11?) and it works fine in all terminals including newest xterm, so why remove support from main "xterm" entry and make it usable only for actual xterm?

I work in enterprise environments, right now my work desktop is Tarantella on RHEL6, that gnome-terminal calls itself xterm and will continue to do so for years (admittedly, I don't log from there to any Fedora 28 host... yet!).

Not only that - manually setting TERM to anything other than xterm/vt100 means quarter of the systems won't recognize it :)

That's why I'm very strongly for keeping "xterm" terminfo entry as conservative/compatible as possible.

If possible, please make the xterm terminfo entry in current ncurses backwards-compatible.

Comment 1 Miroslav Lichvar 2018-05-02 09:08:23 UTC
Upstream has an entry in their FAQ about other terminals emulating xterm, which I think we should follow. Other terminals like urxvt don't have a problem using their own terminfo.

https://invisible-island.net/ncurses/ncurses.faq.html#xterm_generic

The xterm entry should work with recent versions of xterm spanning at least few Fedora releases. The ncurses-base package has other terminfo descriptions like xterm-old, which should work with much older versions of xterm.

I'm not sure which change in ncurses-6.1 broke mouse in vte terminals, but if it works with the current and older versions of real xterm, it's probably another bug in the vte's emulation of xterm.

Comment 2 Egmont Koblinger 2018-05-02 10:24:24 UTC
(In reply to Leszek Matok from comment #0)

> [...] but found out all
> vte-based terminals (lilyterm, termit, lxterminal...) produce strange
> results.
> 
> I work in enterprise environments, right now my work desktop is Tarantella
> on RHEL6, that gnome-terminal calls itself xterm and will continue to do so
> for years

Could you please clarify your setup? As far as I understand, you are _not_ running gnome-terminal (or other vte-based terminals) on the same Fedora 28 box that has ncurses-6.1, correct? Sounds to me you're ssh'ing from RHEL6 to F28. What's the version of your gnome-terminal and vte on the box where you run it?

(In reply to Miroslav Lichvar from comment #1)

> Upstream has an entry in their FAQ about other terminals emulating xterm,
> which I think we should follow. Other terminals like urxvt don't have a
> problem using their own terminfo.

It sounds to me that OP has a problem mixing a brand new ncurses with a probably quite old gnome-terminal. Due to the lack of versioning possibility in terminfo, gnome-terminal (or vte) shipping its own terminal description could not prevent these kinds of issues from happening.

Comment 3 Leszek Matok 2018-05-02 10:37:47 UTC
First a bit of a reply to that FAQ:

It's strange how just one line from what you linked, they say there's "dtterm" info for dtterm. I use CDE's dtterm, it by default DOES set TERM=dtterm but... there's no terminfo for dtterm in current RHEL or Fedora, including this new version. So all the good advice from Thomas's page is incorrect, as I unwillingly have to export TERM=xterm anyways!

(I'm working with old HP-UX, AIX, Solaris and Linux, often versions from another decade, so yes...)

I understand the arguments ncurses (Thomas) makes about differences between different terminal emulators all calling themselves "xterm", but the reality is that the current TERM environment variable's possible values and terminfo database is handled backwards for so many years...

The whole terminal handling should have support for querying features for years (in fact there are codes to ask for "user agent" but that would lead to just another database which would still not support all software. The only correct way is asking for feature support).

But lack of this, and two decades of setting TERM=xterm under rxvt, dtterm, putty, gnome-terminal etc. etc. means that TERM=xterm is the de-facto "compatibility" standard, having little connection with a program called xterm for years...


So that was me arguing that link, but now to the point of this bug:

The change in ncurses 6.1 only REMOVED support for older/other software, because xterm's (the actual one's) mouse handling was correctly supported with old terminfo just fine.

The change in question removed one of two different possibilities of RECEIVED input codes. It's a different argument than choosing "what to send", in which case there can be only one answer. When receiving, you can be liberal and accept all known codes (unless they conflict) and that's what ncurses up to 6.0 was doing.

So to me this looks like just cutting off the compatibility stuff, not because it conflicted with anything. Now, after reading that link, this looks like an act of spite ;)

I'm arguing this is in fact ncurses' regression.

Comment 4 Leszek Matok 2018-05-02 10:45:01 UTC
(In reply to Egmont Koblinger from comment #2)

> Could you please clarify your setup? As far as I understand, you are _not_
> running gnome-terminal (or other vte-based terminals) on the same Fedora 28
> box that has ncurses-6.1, correct? Sounds to me you're ssh'ing from RHEL6 to
> F28. What's the version of your gnome-terminal and vte on the box where you
> run it?

Not yet - I'm reporting this from my home desktop. I mentioned terminal emulators using vte that are in Fedora 28 and exhibit this problem, and pasted exact vte version from F28. You can for example dnf install lilyterm and confirm this behavior on any F28 system.

But even if someone patches the old vte in F28, this won't help the legacy stuff we're stuck with for years.

> It sounds to me that OP has a problem mixing a brand new ncurses with a
> probably quite old gnome-terminal. Due to the lack of versioning possibility
> in terminfo, gnome-terminal (or vte) shipping its own terminal description
> could not prevent these kinds of issues from happening.

This is exactly what I'm saying, lack of versioning brought this on us :)

Comment 5 Egmont Koblinger 2018-05-02 10:50:31 UTC
As for the philosophical part, I'm pretty much on your side, but it's probably way beyond the scope of this bugreport to design (let alone implement) something better.

Alas, within this bad design, apparently the xterm descriptor struggles to find a balance between new features and legacy compatibility.

> The change in ncurses 6.1 only REMOVED support for older/other software,
> because xterm's (the actual one's) mouse handling was correctly supported
> with old terminfo just fine.

I haven't taken a closer look, but I don't think this is correct. It probably ADDED support for a newer mouse protocol that works even beyond row/column 223, which is not yet supported by your gnome-terminal.

Why it needed to do so by breaking compatibility, when the new mouse protocol can co-exist with the old one (and app might request to enable the extension, and parse whichever type of response it receives, depending on whether the terminal emulator supports the extension or not) is beyond my understanding.

Comment 6 Miroslav Lichvar 2018-05-02 10:52:13 UTC
(In reply to Leszek Matok from comment #3)
> It's strange how just one line from what you linked, they say there's
> "dtterm" info for dtterm. I use CDE's dtterm, it by default DOES set
> TERM=dtterm but... there's no terminfo for dtterm in current RHEL or Fedora,
> including this new version. So all the good advice from Thomas's page is
> incorrect, as I unwillingly have to export TERM=xterm anyways!

That would be a good RFE for ncurses to include the dtterm description.

> (I'm working with old HP-UX, AIX, Solaris and Linux, often versions from
> another decade, so yes...)

Should the terminfo descriptions never be updated because someone might have to use them with such an old version of a terminal, which is not even a real xterm, but rather an incomplete emulation? Have you tried it with TERM=xterm-old?

In any case, you can always use the tic tool to compile your own terminfo in $HOME/.terminfo.

Comment 7 Egmont Koblinger 2018-05-02 10:55:08 UTC
(In reply to Leszek Matok from comment #4)

> Not yet - I'm reporting this from my home desktop. I mentioned terminal
> emulators using vte that are in Fedora 28 and exhibit this problem, and
> pasted exact vte version from F28. You can for example dnf install lilyterm
> and confirm this behavior on any F28 system.

In that case the version of legacy vte is irrelevant for gnome-terminal, it's vte291 (which should be version 0.52) that matters. And TERM should be set to xterm-256color by default, is that the case? A few vte-based emulators, such as lilyterm or lxterminal _might_ be using the ancient vte-0.28, we'd need to check the package's details.

I don't have Fedora, I'm running Ubuntu 18.04 which also ships ncurses-6.1 and vte291-0.52, and I cannot reproduce your problem. Mind you, Ubuntu compiles mc against slang, whereas Fedora switched to ncurses not long ago (not sure if reverted since). But I couldn't reproduce any problem with htop either. I'll compile mc against ncurses later on to try.

Comment 8 Leszek Matok 2018-05-02 13:35:27 UTC
Yes, as I've mentioned in the original report, this affects vte-0.28.2-23.fc28.x86_64 (NOT vte291) and they all set "xterm" as terminal, and even if they did set "xterm-256color" that doesn't fix the mouse issue with that old term.

I took the angle that since all those are still included in Fedora (and I still use one), they should still be supported and work out of the box.

repoquery --whatrequires vte shows several packages, even though I doubt gnome-mud or gnurobots need mouse support in the emulated terminal ;)

I found older discussion about this in Arch, https://bugs.archlinux.org/task/57676 and their solution worries me - I hope Fedora won't resolve to removing the last working gtk2 terminals from us :)

Comment 9 Egmont Koblinger 2018-05-02 13:44:07 UTC
(In reply to Leszek Matok from comment #8)

> Yes, as I've mentioned in the original report, this affects
> vte-0.28.2-23.fc28.x86_64 (NOT vte291) and they all set "xterm" as terminal,
> and even if they did set "xterm-256color" that doesn't fix the mouse issue
> with that old term.

I see. (I got confused by comment 4.)

> I found older discussion about this in Arch,
> https://bugs.archlinux.org/task/57676 and their solution worries me - I hope
> Fedora won't resolve to removing the last working gtk2 terminals from us :)

If you had a vague idea what kinds of bugs and how many were there in vte-0.28 that are fixed since then, you'd never want to look back at them. Privacy/security concerns (scrollback written to disk unencrypted), data corruption in the scrollback, terrible performance, broken emulation at many different places, let alone all the new features that weren't present back then... including the one that causes trouble for you right now. vte-0.28 is long abandoned by mainstream and will eventually vanish from distros, you can't stop this from happening. I'm disappointed it hasn't happened by now. E.g. the current bug is one big reason to compile all the vte-based apps against vte291, or drop the ones that don't support it.

Comment 10 Leszek Matok 2018-05-02 14:08:15 UTC
Yes, well, too bad vte291 needs gtk3 then. I read somewhere they're planning to stabilize gtk3, so maybe one day I will move on indeed! Will take years at this pace though.

Meanwhile, however, that old vte (except the missing features I have to disagree, especially performance-wise, vte-0.28 shines :)) is a piece of software included in the Fedora Collection, so it should be supported :)

So we're back to the question whether it's possible to re-add support of the old 223-line mouse protocol in terminfo for xterm.


PS. Fedora switched mc back to slang immediately, as everyone's function keys broke with ncurses (Shift-F6 etc.) - but it's still affected by terminfo updates which come with ncurses packages.

Comment 11 Egmont Koblinger 2018-05-02 19:51:48 UTC
(In reply to Leszek Matok from comment #10)

> except the missing features I have to
> disagree, especially performance-wise, vte-0.28 shines :)

Check its git log, open the linked bugreports... As for performance, if you cat a giant file, it's about 10x slower than vte291, probably by far the slowest terminal emulator shipped by Fedora.

> Meanwhile, however, that old vte [...] is a piece of
> software included in the Fedora Collection, so it should be supported :)

And that's where my suggestion for Fedora developers is to drop these from the next release :)

> So we're back to the question whether it's possible to re-add support of the
> old 223-line mouse protocol in terminfo for xterm.

Even though the two protocols can nicely live together: an app could without any problem switch to the new mode if available, but fall back to the old one if not; apparently terminfo is not capable of describing this autodetection (and/or ncurses cannot autodetect the mode this way), it needs to encode either one or the other. This is a limitation of ncurses/terminfo.

Switching back to the old one would probably cause regression (compared to current F28) for way more people than switching forward caused (I doubt lilyterm, termit, lxterminal have significant user bases), hence I do not advocate your proposal.

Probably a much better approach is to backport the mouse extension into vte (and in the mean time backport a few other important and simple changes, such as adding REP support also needed for new terminfo's xterm, fixing bracketed paste which often causes problems with mc, the one-liner fix resulting in the 10x speedup, etc.).

> PS. Fedora switched mc back to slang immediately, as everyone's function
> keys broke with ncurses (Shift-F6 etc.) - but it's still affected by
> terminfo updates which come with ncurses packages.

Thanks for this info, I knew it had troubles but didn't follow whether the change got reverted.

Comment 12 Egmont Koblinger 2018-05-02 19:53:30 UTC
(In reply to Miroslav Lichvar from comment #1)

> Component: ncurses → vte291

As per followup comments, the affected packages are ncurses/terminfo and legacy vte, but not vte291.

Comment 13 Leszek Matok 2018-05-03 20:48:43 UTC
So I went ahead and patched VTE 0.28.2 with your extended mouse tracking patches as well as that one-liner speedup and hope to get this into Fedora as Bug 1574683 - we'll see what the maintainer thinks.

(RHEL6's gnome-terminal has that performance patch backported as well, maybe that's why I never noticed ;))


However, this bug remains to see whether it's possible for terminfo to still support old terminals' mouse tracking without the newer 1006 extension (for the possibility of people logging from like RHEL7 client to RHEL8 server some time from now).

Comment 14 Egmont Koblinger 2018-05-04 09:13:21 UTC
(In reply to Leszek Matok from comment #13)

> (RHEL6's gnome-terminal has that performance patch backported as well, maybe
> that's why I never noticed ;))

Fair enough :)

> However, this bug remains to see whether it's possible for terminfo to still
> support old terminals' mouse tracking without the newer 1006 extension (for
> the possibility of people logging from like RHEL7 client to RHEL8 server
> some time from now).

I still believe that supporting clicks beyond column 223 out of the box in modern terminal emulators is a much more important use case than yours. (The decision is of course Red Hat / Fedora developers'. I'm not affiliated with them.)

Comment 15 Leszek Matok 2018-05-04 09:55:07 UTC
(In reply to Egmont Koblinger from comment #14)

> I still believe that supporting clicks beyond column 223 out of the box in
> modern terminal emulators is a much more important use case than yours.

Oh, of course!

Only in a case when it's possible to support both, it should be preserved.

(But on first glance it appears it's possible and was in place previously, now removed as obsolete or something.)

Comment 16 Thomas E. Dickey 2018-10-20 16:06:23 UTC
I just noticed this; however aside from some remaining issues reported in

https://bugzilla.redhat.com/show_bug.cgi?id=1610681

I don't see any comments/points that are useful:

a) the report does not clarify _how_ the Fedora package impacts the RHEL 6
   system.  If OP was logging into a Fedora 28 system from RHEL 6 (not the
   usual case...), that should have been mentioned up front.

b) the package is for Fedora 28, not RHEL 6.
   The latter doesn't have ncurses 6.1

c) there were fixes made in slang to handle ncurses 6.1's terminfo updates.
   Since this report doesn't go into any detail, it's worth pointing out
   that those issues were fixed a few months before this report.

d) there have been no useful bug-reports about function-keys (that part's
   been fairly stable -- even back to RHEL 6).

e) ncurses has had a dtterm entry since 1996.  Whether or not Redhat's
   package includes it is not related to upstream ncurses.

Comment 17 Miroslav Lichvar 2018-10-23 07:49:56 UTC
(In reply to Thomas E. Dickey from comment #16)
> e) ncurses has had a dtterm entry since 1996.  Whether or not Redhat's
>    package includes it is not related to upstream ncurses.

Thanks. I missed that. It's included in the ncurses-term package (not the ncurses-base package).

Comment 18 Ben Cotton 2019-05-02 20:36:05 UTC
This message is a reminder that Fedora 28 is nearing its end of life.
On 2019-May-28 Fedora will stop maintaining and issuing updates for
Fedora 28. It is Fedora's policy to close all bug reports from releases
that are no longer maintained. At that time this bug will be closed as
EOL if it remains open with a Fedora 'version' of '28'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 28 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 19 Ben Cotton 2019-05-28 22:59:25 UTC
Fedora 28 changed to end-of-life (EOL) status on 2019-05-28. Fedora 28 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.