Note, that in RHEL9, your scp is using SFTP protocol in the background, which is a bit better defined than the SCP.

I would say this is expected. Doing this is at least racy or more like undefined behavior when you try to read and write one file at the same time. It worked with SCP in RHEL8 probably by pure luck.

I am really not sure what intention would this use case have in the first place, but if you need to preserve this hack for some reason, please use the -O switch, which will use the SCP.

Thank you for the clarification and let's hope others will find this BZ when looking for their broken scripts. As for the why are we using it, it's used by pbench to deploy testing scripts where the clients can be remote as well as local. Pbench's is_remote detection is very simple ($hostname == localhost || 127.0.0.1) but a fix based on python is coming.

Unfortunately, this is an upstream bug and has no chances to be fixed soon. I think we should have to document this behaviour as an incompatibility.

*** Bug 2082440 has been marked as a duplicate of this bug. ***

------- Comment From christof.schmitt.com 2022-05-06 12:42 EDT-------
(In reply to comment #10)

> Thank you for the clarification and let's hope others will find this BZ when
> looking for their broken scripts. As for the why are we using it, it's used
> by pbench to deploy testing scripts where the clients can be remote as well
> as local. Pbench's is_remote detection is very simple ($hostname ==
> localhost || 127.0.0.1) but a fix based on python is coming.

This affects IBM Spectrum Scale, and i am guessing likely other cluster solutions
as well. It might not be ideal, but for a long time, scp to all cluster nodes,
including the local one was possible. My concern is that this will break
more systems. Working around the changed behavior can also be risky.

> Unfortunately, this is an upstream bug and has no chances to be fixed soon.
> I think we should have to document this behaviour as an incompatibility.
>
> *** Bug 2082440 has been marked as a duplicate of this bug. ***

Looking at
https://www.openssh.com/txt/release-8.9
|Future deprecation notice
|=========================
|
|A near-future release of OpenSSH will switch scp(1) from using the
|legacy scp/rcp protocol to using SFTP by default.

So upstream OpenSSH has not switched to sftp behind scp yet. Today,
this bug only affects RHEL9, so the statement that this is an
"upstream bug" is not correct.

Switching protocols in the scp backend is probably a good improvement,
but this should never introduce a behavior change. This should be fixed
before RHEL9 GA, to avoid breakage in scp users.

The switch has happened in 9.0 (upstream) which is already released:
https://www.openssh.com/txt/release-9.0
so the statement is correct.

Unfortunately, the behavior change when switching the protocol was inevitable and major version change is a natural place to change it. If you want to preserve the old behavior, you still may enforce using the old protocol via -O key, though it is not recommended.

------- Comment From christof.schmitt.com 2022-05-06 16:46 EDT-------
(In reply to comment #12)
> The switch has happened in 9.0 (upstream) which is already released:
> https://www.openssh.com/txt/release-9.0
> so the statement is correct.
>
> Unfortunately, the behavior change when switching the protocol was
> inevitable and major version change is a natural place to change it. If you
> want to preserve the old behavior, you still may enforce using the old
> protocol via -O key, though it is not recommended.

Apologies for missing OpenSSH 9.0. The change that results in losing data
is very unfortunate. Luckily, we discovered this early for our product, but
i would expect that other RHEL9 users will run into the same problem. One
could argue that this is a "data loss" problem, that can easily be triggered.

This is a ML thread about this issue:

https://lists.mindrot.org/pipermail/openssh-unix-dev/2022-February/040056.html 

Unfortunately upstream considers it as a corner case. It maybe worth raising an issue in OpenSSH Bugzilla (https://bugzilla.mindrot.org/)

------- Comment From christof.schmitt.com 2022-05-06 16:54 EDT-------
The post at https://www.redhat.com/en/blog/openssh-scp-deprecation-rhel-9-what-you-need-know
documents the change, but it does not call-out that overwriting a file will
now discard data. All of that is a big surprise, and any script that just
calls scp has the risk of losing data.

------- Comment From christof.schmitt.com 2022-05-06 17:03 EDT-------
(In reply to comment #15)
> This is a ML thread about this issue:
>
> https://lists.mindrot.org/pipermail/openssh-unix-dev/2022-February/040056.
> html
>
> Unfortunately upstream considers it as a corner case. It maybe worth raising
> an issue in OpenSSH Bugzilla (https://bugzilla.mindrot.org/)

Can Redhat open a bugzilla for OpenSSH? From IBM, we can add a comment that this
is a data-loss issue.

------- Comment From christof.schmitt.com 2022-05-10 18:22 EDT-------
Opened OpenSSH bugzilla:
https://bugzilla.mindrot.org/show_bug.cgi?id=3431

I still think that losing data is the worst possible outcome here.
Even just returning an error would be much more user-friendly, than
having users have to restore their data again.

------- Comment From christof.schmitt.com 2022-05-12 12:15 EDT-------
The question about using the -O parameter to fallback to the scp protocol
came up. We don't see the -O parameter as a sufficient workaround:

1) The -O flag does not exist on older scp versions, so  a script can longer just call scp, but it needs to start with version checks.
2) The admin can decide to create the file /etc/ssh/disable_scp to complete disable the scp protocol, which also disables this workaround.
3) Apparently the fallback to the scp protocol will also be completely removed eventually, so we could not rely on this forever.

I assume that switching out the protocol behind the scp tool is the appropriate thing to do. All we are asking is to fix this one cornercase, to not silently discard data.

https://bugzilla.mindrot.org/show_bug.cgi?id=3431 is the upstream fix

------- Comment From christof.schmitt.com 2022-05-13 13:59 EDT-------
Thank you for pointing out the upstream fix!

This is still an urgent issue for us with RHEL9.
When will this fix be available in RHEL9? Could we
get an ETA for our planning?

Would it be possible get a RHEL9 test package of
openssh with the fix applied, so that we can verify
the fix?

------- Comment From seg.com 2022-07-25 17:51 EDT-------
Also, note that the related RH KB article doesn't help. An actual Z stream fix is urgent and essential.

We understand your reasons and are working on this bug including Z stream. When it is fixed, we will immediately let you know

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (openssh bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:8375