Done in rawhide (F36)

I think this may be causing problems. I recently upgraded to Rawhide, and trying to scp from an F34 host fails:

$ scp openqa01.iad2.fedoraproject.org:/var/lib/openqa/share/tests/fedora/needles/*.json .
subsystem request failed on channel 0
Connection closed

with -vvvv, this is where things die:

debug1: Sending subsystem: sftp
debug2: channel 0: request subsystem confirm 1
debug3: send packet: type 98
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug3: receive packet: type 100
debug2: channel_input_status_confirm: type 100 id 0
subsystem request failed on channel 0
Connection closed

oh, note that the connection is proxied, these are the relevant lines in ~/.ssh/config:

Host [redacted]
    User [redacted]
    IdentityFile [redacted]
    HostName %h
    ProxyCommand ssh -q [redacted].fedoraproject.org /usr/bin/nc %h 22

Indeed, using 'scp -O' works.

(In reply to Adam Williamson from comment #3)
> oh, note that the connection is proxied, these are the relevant lines in
> ~/.ssh/config:
> 
> Host [redacted]
>     User [redacted]
>     IdentityFile [redacted]
>     HostName %h
>     ProxyCommand ssh -q [redacted].fedoraproject.org /usr/bin/nc %h 22

Why do you use netcat for IO forwarding? Is there a specific reason not to use just ProxyJump or ProxyCommand="ssh -W %h:%p Jumphost"?

In any case, if you remove the -q you will get some more logs why the connection failed.

Cos that's the magic blob I got from the infra team :)

nirik, is the above still what I should be using? If not, then what?

So, our nc ProxyCommand predates ssh having ProxyJump. ;) 
   ProxyJump yourusername.org 
should be fine. 

On sftp, we disable it by default. 

You want to set: 

sshd_sftp: true

in inventory/group_vars/openqa* most likely to enable it.

I don't particularly care what protocol it's using, I just want it to work without bugging me :D I would assume that it should gracefully downgrade in the case where sftp isn't available somewhere along the chain.

So I commented out the ProxyCommand line and changed it to:

ProxyJump adamwill.org

but the bug still happens the same. The end of the -vvvv output is now:

debug1: Sending subsystem: sftp
debug2: channel 0: request subsystem confirm 1
debug3: send packet: type 98
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug3: receive packet: type 100
debug2: channel_input_status_confirm: type 100 id 0
subsystem request failed on channel 0
debug3: send packet: type 1
debug1: channel 0: free: direct-tcpip: listening port 0 for openqa01.iad2.fedoraproject.org port 22, connect from 127.0.0.1 port 65535 to UNKNOWN port 65536, nchannels 1
debug3: channel 0: status: The following connections are open:
  #0 direct-tcpip: listening port 0 for openqa01.iad2.fedoraproject.org port 22, connect from 127.0.0.1 port 65535 to UNKNOWN port 65536 (t4 r0 i0/0 o0/0 e[closed]/0 fd 4/5/-1 sock -1 cc -1)

Killed by signal 1.
Connection closed

> subsystem request failed on channel 0

Thats the sftp subsystem request that failed. I am not sure if the fallback to scp was implemented in the end. Dima can confirm.

So I think the first solution for you would be to enable sftp on the target host. The middle jumpbox does not care as it just forwards IO. It should solve you some headaches of scp, but you can still use muscle-memoriezed scp commands.

Thanks. Still, I really think we should consider making fallback work here. Otherwise you are gonna be *flooded* with instances of this problem; this situation is not going to be at all uncommon, I don't think.

I'm not sure fallback is a good idea. When breaking a backward compatibility, it should be as explicit as possible :)

If there's no fallback, a much better error message - explaining exactly what the issue is, and how to address it - would be a very good idea to avoid a large amount of unnecessary explaining things to frustrated people.