Bug 1017290

Just for completeness, I'm easily able to reproduce the issue on RHEL6 RPM distro accessed through ssh in a X server terminal emulator.

hm, this is something ive tried to test without being able to reproduce. could you please explain which machines and possibly machines i can access as well to try to debug this?

Ståle, I can always reproduce on EC2 machines run in us-east-1 region and ssh-ing to them from europe. I'm not sure where you are located but it's possible that connection speed and/or rtt does make a difference.

If you email me your public ssh key and tell me your location, I can start an istance to try it out.

ive been in contact with Aleksandar and he provided access to an ec2 instance.
i tried to reproduce what he has seen, but ive been unsucessful.
ive tested from ubuntu and f19 with not ssh settings and with Forward Agent.
are anyone else able to reproduce this?
- im really interested in fixing issues like this, i just need a way to reproduce it.

Created attachment 810489 [details]
screenshot of corruption

Hello again, it's very interesting that you can't reproduce. I logged in to the same machine and tried with the aesh version you used. I did reproduce the problem. See the attached screenshot that I took after holding left and right arrow keys a few times. I basically entered a line with random latin characters but after keeping arrows pressed a few times you can see I have some weird characters in the line.

I'll try a different terminal program as well I'll try a different font but if you have any better idea how to debug, please let me know.

well, ive enabled some extra logging in the aesh client. so, before you start, remove all /tmp/aesh.log* files.
start aesh again, and look for the "GOT:" lines, that will print out what kind of input values are given.
if you can find out what is sent to aesh when you press left/right arrow, that would be nice.

- just to be sure, aesh/jboss-cli works as expected on your local machine?

Created attachment 810506 [details]
aesh log when corruption happens

Attaching aesh log after I experienced a corruption. What I did is:
1. enter random string
2. hold left arrow for a few seconds
3. hit enter
4. quit, enter

And yes, I do not experience the problem locally. It seems to be timing related in any case.

FYI from a single run two log fiels were produced - aesh.log and aesh.log.1. aesh.log contains only this:
> Oct 10, 2013 8:59:25 AM org.jboss.aesh.console.Config parseInputrc
> INFO: Error while parsing: /root/.inputrc couldn't find file.

thanks for the log. yes, because of some sort of lag the arrow value (int[3]) is sent twice in one "packet". eg its now a int[6].
i should be able to handle this. ill keep you updated.

Aleksandar, ive pushed a new version of æsh to the amazon instance. could you test that? it should try to ignore double/triple inputs that is caused by lag.

I tried a few things with the new version. I don't see corruption anymore when pressing arrow keys. But I could break it in two other ways:
1. go to middle of string and paste something - only part of the pasted string appears on screen
2. try deleting from middle of string - weird characters appear on line

Please see the two new screenshots I'm attaching in a moment as well the log from last run.

Created attachment 810974 [details]
aesh log - corruption from pasting and backspace

Created attachment 810975 [details]
screenshot after pasting a line with middle mouse click

Created attachment 810976 [details]
screenshot after pressing backspace

such a big log is unfortunately not very useful when i do know what kind of keys you press at the same time of the output, but from what i can see the backspace error is identical to the arrow issue, which ill try to fix.

what happens with paste i cant answer other than that im hoping it is related to the backspace.
ill create a fix for backspace as well...

Created attachment 811930 [details]
screenshot of corruption from pressing a letter key

Hello again. It turns out any key press can cause some kind of corruption. Now I tried hitting a letter key. which results in the attached screenshot. I'll attach as well a log file. What I did is type a random string spanning on 3 lines. Go with arrow keys to first line and hit `g`. When corruption occurred I hit enter and then quit.

Isn't it possible to handle such lag situations universally without losing key-presses and without recognizing wrong keys?

Created attachment 811931 [details]
aesh log - corruption after letter key pressed

hi, thanks for the log/screenshot. they way æsh works now is that it has a separate thread that reads from the inputstream. the threads sleeps for 10ms between each poll. this have seemed to work very well (until now), so im a but reluctant to change how that works tbh since it seems like the input is coming in chunks when you're testing it and reducing the sleep should not have that much of an impact.
ill look into it.

i did some changes to the reader thread upstream and ported it back to this branch hoping it might help a bit.
ive uploaded a new version of aesh, could you take it for a spin please? :)

Where is the updated version? I see only the previous one - aesh-0.33.10-SNAPSHOT.jar

And I broke it for a second with the backspace where it spits out on stdout (not log) a lot of these lines:
> at org.jboss.aesh.edit.KeyOperationManager.findOtherOperations(KeyOperationManager.java:101)

I couldn't get the full trace because of terminal buffer. But if this is the right aesh version to test with, I'll do something to get it.

ah, sorry, the jar i pushed contained a bug. ive pused a new version now, its the same jar name (aesh-0.33.10-SNAPSHOT.jar).

See /tmp/aesh.log.1

I enetered a long line of text then wrote moderately fast "I want to break free" and saw line corruptions. It's harder to reproduce though, it happened from second attempt. My first attempt was successful with only one probelm. I didin't see any corruption artifacts but everything is very slow. Additionally when I wrote "I want to break free", I only got "Iwn  bea fr" in console.

It's very interesting why can't you reproduce this though, I can't understand it. I don't have any problems with command line outside aesh.

ive been trying to reproduce this on the same server several times, but i have not been successful (i would prefer not to bug you every time :)
ive been trying to write as fast as i can, holding a key in for a long time, pasting a rather long line and then start typing, but i havent been able to reproduce it yet unfortunately.

i realize that this is not 100% fixed, and i think i could change more in æsh to make it better/fix it completely, but for now i do not have much more time to work on this until in november i would think.

for now, im voting for releasing 0.33.10 and that we use that for eap6.2. atleast it fixes the issue with garbage text in the terminal (if not 100%, then close to it).

I did additional investigation trying to reproduce the issue locally. Here I'll list my observations for future reference. I used the uploaded aesh 0.33.10 snapshot.

First of all I couldn't reproduce any issue running on local machine only. No matter what I tried.

I created a virtual machine [1] so I go through a network and ssh. Without touching anything from defaults I was able to reproduce one issue - when in the beginning of a long line (spanning at least 3 lines of the terminal) [2] keeping backspace pressed causes sometimes different kinds of corruption. I did various things to have a stable reproducer but nothing really appears stable. I still need a couple of attempts and fiddling with the command line to get something. What I found though is that some settings seem to make reproducing easier.
* using xfce terminal emulator makes it easier to reproduce than konsole or plain linux text console (still I could reproduce on all of them)
* inserting network delay of 200-300ms between machines somehow helps to reproduce [3]; delay can be set on the virtual machine eth0 and host machine vnet# interface
* increasing key repeat rate from desktop environment settings somehow increases the chances of getting a corruption
* having the terminal screen full with data so that you are operating on the last line helps get a corruption
* one may need to enter multiple lines, call from history lines with different lengths before an issue is seen

I'm really out of ideas how can I improve reproducibility.

--------

Another interesting observation I have is that aesh cli is very slow. Much slower than native cli console over ssh. Especially that's visible when deleting from the beginning of a long line (spanning at least 3 lines of the terminal). When there is network delay, it is even slower. Unlike native cli where key presses are delayed the specified time, but after that operations execute with the same speed/throughput.
It's strange to see how cursor is jumping while the delete key is pressed. Not sure but this could have something to do with the problems.



I hope my explanation is understandable. Let me know if something is unclear or if you want more details about virtual machine or netem. I think this way to reproduce should be more reliable than the EC2 machine (which I'm going to stop for now). I was running on core i7 with lots of free ram and negligible CPU load so my guess is the VM performance wouldn't be much slower than on your hardware whatever it is.



[1] basically installed fedora 19 on my fedora 18 using virt-manager and netinst iso, this link is relevant: https://fedoraproject.org/wiki/Getting_started_with_virtualization
[2] example string I was using: 3kd hkdf hgkfdhj gkjfd gkdj gfkdfgkdk fdkd kd hgkjd gfkd gfkd gkdhfk ghkd gkdj gkd gkfdh kd hgkdf gkd fkd hkd kdf hgkgkdhfkdkdf hkdfj hkd hfkd hkdf hkdf hkd hfkdf hjkdjf kfdj kfdkfd  fdkdjf gkdj kd gkdf gkdf kfdh kfd gkdgkdhkdf hkfd kfd gkd kfd gkfd gkfd gkdf hkdhfk dhkfd hk fdh"
[3] http://purefinity.blogspot.com/2009/01/simulating-network-delay-using-linux.html

thanks for the thorough investigation.
i think i know what is going on; æsh receives the input in "groups" instead of one and one input character (as expected). æsh could most likely be enhanced to parse this better, and i have a jira for it as well, but it wont be prioritized yet im afraid.

regarding the observation that æsh is slow, i agree it is slower than bash when doing deletion of a biiiiig text where you delete from anywhere except the end. æsh could do this smarter and more efficient, but atm it tries to do it correctly. the latest version of æsh does it more efficiently, but there are still optimizations to be made (which ill do when time permits). still, i dont see it as vital since this is not common cli behaviour. most commands are 1-3 lines long, so this would be a rare use case.

FYI I finally managed to reproduce locally using a virtual fedora 19 machine. In my first attempts I only added latency but this is not enough. Packet reordering seems to be needed. Basically the steps to reproduce are:
* create a virtual machine (virt-manager does the job just fine)
* run the virtual machine and enable ssh access to it
* add netem queuing discipline to the virt network device connecting to the VM (e.g. vnet0)
** sudo tc qdisc add dev vnet0 root netem delay 100ms reorder 50% 50%
* ssh to the virtual machine
* java -cp aesh-0.33.10-SNAPSHOT.jar Example
* type letters with one hand and at the same time repeatedly hit backspace with the other hand.

Result would be some weird characters in the terminal. As well it is obvious some backspace repeats are ignored if backspace key is being pressed some periods of time (e.g. a few seconds).

I didn't notice any difference whether or not I add packet reordering in the reverse direction (i.e. VM->host). If you want to try that as well, it can be done from the VM itself like:
* sudo tc qdisc add dev eth0 root netem delay 100ms reorder 50% 50%

Please let me know if I can give further details.

hi, i rewrote how æsh parses input in æsh 0.43, could you please test it and see if works better?
you can download it from here: https://github.com/aeshell/aesh/archive/0.43.tar.gz

I was able to reproduce lost key presses on a network machine. Will try to reproduce locally tomorrow. But what does it mean that the prompt turns into `[FOO]»` instead of `[test]$` ?

hm, if there are no actual missing bytes there should not be any missing keys anymore :/
the prompt changes after 4 seconds just to show how the async stuff can be used.

Created attachment 835320 [details]
aesh gnu script log

I can't understand what do you mean with no missing bytes. But I recorded a short `script` session to visualize to you what is going on. It is based on comment 26 for that I was typing only alphanumeric characters with one hand and repetitively hitting backspace with the other hand.
To reply that you do `scriptreplay aesh.time.log aesh.log`. You will see the weird character output in the line. Unfortunately this can serve only for illustration purposes because I can't yet get both - input and output log. I'm trying to find sources of NetBSD screen because it is claimed to have that functionality.. or I may try running it in a VM.

But perhaps it can be easier if you can use a second machine or a VM to reproduce locally.

Sorry, I can't compile netbsd's `script` in fedora as well netbsd instances I'm trying to run in EC2 are not starting correctly. So only option seems to be to reproduce with netem and another machine or a virtual machine.

This issue is reported against a version which is no longer maintained, which means that it will not receive a fix. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of Enterprise Application Platform please feel free to open a bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.