Bug 2038010

Do you run the scp on the host to copy file between guests?

Did you try to ping both guests from host using "ping -6"?

Could you provide the result of "ip a" and "ip r" on host and guests?

Could you provide the result of "ip -6 neigh" and "ip -4 neigh" on the host?

(In reply to Laurent Vivier from comment #2)

Hi Laurent

> Do you run the scp on the host to copy file between guests?
Yes, transfer data from host to vm1/vm2 test pass.

> 
> Did you try to ping both guests from host using "ping -6"?
Yes, I tried. ping -6 test pass.

# ping -6 fe80::9895:efff:fe41:832e -c 2 -I switch
ping: Warning: source address might be selected on device other than: switch
PING fe80::9895:efff:fe41:832e(fe80::9895:efff:fe41:832e) from :: switch: 56 data bytes
64 bytes from fe80::9895:efff:fe41:832e%switch: icmp_seq=1 ttl=64 time=0.312 ms
64 bytes from fe80::9895:efff:fe41:832e%switch: icmp_seq=2 ttl=64 time=0.118 ms

--- fe80::9895:efff:fe41:832e ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1004ms
rtt min/avg/max/mdev = 0.118/0.215/0.312/0.097 ms

# ping -6 fe80::9895:efff:fe41:832d -c 2 -I switch
ping: Warning: source address might be selected on device other than: switch
PING fe80::9895:efff:fe41:832d(fe80::9895:efff:fe41:832d) from :: switch: 56 data bytes
64 bytes from fe80::9895:efff:fe41:832d%switch: icmp_seq=1 ttl=64 time=0.309 ms
64 bytes from fe80::9895:efff:fe41:832d%switch: icmp_seq=2 ttl=64 time=0.114 ms

--- fe80::9895:efff:fe41:832d ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1024ms
rtt min/avg/max/mdev = 0.114/0.211/0.309/0.097 ms


> 
> Could you provide the result of "ip a" and "ip r" on host and guests?
Host:
# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master switch state UP group default qlen 1000
    link/ether f4:8e:38:c3:1b:58 brd ff:ff:ff:ff:ff:ff
    altname enp1s0f0
6: switch: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether f4:8e:38:c3:1b:58 brd ff:ff:ff:ff:ff:ff
    inet 10.73.225.40/22 brd 10.73.227.255 scope global dynamic noprefixroute switch
       valid_lft 69034sec preferred_lft 69034sec
    inet6 2620:52:0:49e0:526a:dd6d:5c3:8cd0/64 scope global dynamic noprefixroute 
       valid_lft 2591980sec preferred_lft 604780sec
    inet6 fe80::36e4:79c5:d653:42d8/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
448: tap0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master switch state UNKNOWN group default qlen 1000
    link/ether ee:63:a4:a6:98:79 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::ec63:a4ff:fea6:9879/64 scope link 
       valid_lft forever preferred_lft forever
449: tap1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master switch state UNKNOWN group default qlen 1000
    link/ether 8a:5e:d2:9d:43:07 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::885e:d2ff:fe9d:4307/64 scope link 
       valid_lft forever preferred_lft forever

# ip r
default via 10.73.227.254 dev switch proto dhcp metric 425 
10.73.224.0/22 dev switch proto kernel scope link src 10.73.225.40 metric 425

VM1:
# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 9a:95:ef:41:83:2d brd ff:ff:ff:ff:ff:ff
    altname enp5s0
    inet 10.73.227.46/22 brd 10.73.227.255 scope global dynamic noprefixroute eth0
       valid_lft 84933sec preferred_lft 84933sec
    inet6 2620:52:0:49e0:9895:efff:fe41:832d/64 scope global dynamic noprefixroute 
       valid_lft 2591999sec preferred_lft 604799sec
    inet6 fe80::9895:efff:fe41:832d/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
# ip r
default via 10.73.227.254 dev eth0 proto dhcp metric 100 
10.73.224.0/22 dev eth0 proto kernel scope link src 10.73.227.46 metric 100 

VM2:
# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 9a:95:ef:41:83:2e brd ff:ff:ff:ff:ff:ff
    altname enp5s0
    inet 10.73.227.163/22 brd 10.73.227.255 scope global dynamic noprefixroute eth0
       valid_lft 84889sec preferred_lft 84889sec
    inet6 2620:52:0:49e0:9895:efff:fe41:832e/64 scope global dynamic noprefixroute 
       valid_lft 2591958sec preferred_lft 604758sec
    inet6 fe80::9895:efff:fe41:832e/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
# ip r
default via 10.73.227.254 dev eth0 proto dhcp metric 100 
10.73.224.0/22 dev eth0 proto kernel scope link src 10.73.227.163 metric 100 

> 
> Could you provide the result of "ip -6 neigh" and "ip -4 neigh" on the host?

# ip -6 neigh
fe80::9895:efff:fe41:832d dev switch lladdr 9a:95:ef:41:83:2d REACHABLE
fe80::9895:efff:fe41:832e dev switch lladdr 9a:95:ef:41:83:2e REACHABLE
fe80::46aa:5000:8ccc:2ba1 dev switch lladdr 44:aa:50:cc:2b:a1 router STALE

# ip -4 neigh
10.73.227.254 dev switch lladdr 44:aa:50:cc:2b:a1 REACHABLE

Best Regards
Lei

Hello Laurent

One thing I need to clarify, Link-local IPv6 address is a local unicast address with prefix fe80 which is the default ipv6 address of the mac address. 
The address prefixed with "2620:" is the ipv6 address configured by the IT department on the lab switch for different mac addresses.

So, is the above test scenario a product bug?  If it's a product bug, does QE also need to test addresses prefixed with "2620:"?

Best Regards
Lei

(In reply to Lei Yang from comment #4)
> Hello Laurent
> 
> One thing I need to clarify, Link-local IPv6 address is a local unicast
> address with prefix fe80 which is the default ipv6 address of the mac
> address. 
> The address prefixed with "2620:" is the ipv6 address configured by the IT
> department on the lab switch for different mac addresses.
> 
> So, is the above test scenario a product bug?  If it's a product bug, does
> QE also need to test addresses prefixed with "2620:"?
> 

I think the problem described in this BZ can come from a misuse of IPv6 syntax with ssh remote to remote copy feature.
It can be impacted by the prefix (local or not) used to address the remote systems.

But I'm neither a IPv6 expert nor a ssh expert so I can't say if it's really the problem and if we need to test a different kind of IPv6 address.

Perhaps you can try to use the "2620:" address to see if the result differs from the "fe80:" one?

Dmitry,

it seems openssh BZ are generally assigned to you: could you help to check the ssh command used in this BZ and to tell if the problem is with the ssh syntax or in the networking stack?

(In reply to Laurent Vivier from comment #5)
> (In reply to Lei Yang from comment #4)
> > Hello Laurent
> > 
> > One thing I need to clarify, Link-local IPv6 address is a local unicast
> > address with prefix fe80 which is the default ipv6 address of the mac
> > address. 
> > The address prefixed with "2620:" is the ipv6 address configured by the IT
> > department on the lab switch for different mac addresses.
> > 
> > So, is the above test scenario a product bug?  If it's a product bug, does
> > QE also need to test addresses prefixed with "2620:"?
> > 
> 
> I think the problem described in this BZ can come from a misuse of IPv6
> syntax with ssh remote to remote copy feature.
> It can be impacted by the prefix (local or not) used to address the remote
> systems.
> 
> But I'm neither a IPv6 expert nor a ssh expert so I can't say if it's really
> the problem and if we need to test a different kind of IPv6 address.
> 
> Perhaps you can try to use the "2620:" address to see if the result differs
> from the "fe80:" one?

Hello Laurent

I tried to use the "2620:" test this sceanrio, transfer file succeed.

Best Regards
Lei

> 
> Dmitry,
> 
> it seems openssh BZ are generally assigned to you: could you help to check
> the ssh command used in this BZ and to tell if the problem is with the ssh
> syntax or in the networking stack?

Test pass with openssh-8.7p1-24.el9_1.x86_64, maybe this bug has fixed in the latest version.

After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.

Hi Laurent

I have to rise this problem again,because the latest rhel 9.4 also can reproduced problem. In the past we have not been able to confirm the root cause of this problem. So I'd like to confirm whether re-open this bug, or it can be removed from QE test plan.This doesn't seem like a widely used scenario.

Thanks
Lei

(In reply to Lei Yang from comment #30)
> Hi Laurent
> 
> I have to rise this problem again,because the latest rhel 9.4 also can
> reproduced problem. In the past we have not been able to confirm the root
> cause of this problem. So I'd like to confirm whether re-open this bug, or
> it can be removed from QE test plan.This doesn't seem like a widely used
> scenario.

What is the version of openssh in rhel-9.4?

Is it again a regression?

In comment #22 (openssh-8.7p1-24.el9_1.x86_64) it seems to have been fixed, and not in comment #26 (openssh-8.7p1-28.el9.x86_64).

I think the problem is with the syntax used with scp (step 3).

I think it's a valid QE test.

Can I have access to your test machine to experiment some solutions?

(In reply to Laurent Vivier from comment #31)
> (In reply to Lei Yang from comment #30)
> > Hi Laurent
> > 
> > I have to rise this problem again,because the latest rhel 9.4 also can
> > reproduced problem. In the past we have not been able to confirm the root
> > cause of this problem. So I'd like to confirm whether re-open this bug, or
> > it can be removed from QE test plan.This doesn't seem like a widely used
> > scenario.
>
> What is the version of openssh in rhel-9.4?

openssh-8.7p1-35.el9.x86_64
> 
> Is it again a regression?
> 
> In comment #22 (openssh-8.7p1-24.el9_1.x86_64) it seems to have been fixed,
> and not in comment #26 (openssh-8.7p1-28.el9.x86_64).

I think this is a regression issues, since it can test pass before I reported this bug(BTW, latest rhel 8 can test pass). But QE can not determine the root cause, because I tried downgrade to openssh-8.7p1-24.el9_1.x86_64, it also can reproduced this problem. 
> 
> I think the problem is with the syntax used with scp (step 3).

Agree with you, it looks like the syntax for using ipv6 scp has changed on rhel 9.
> 
> I think it's a valid QE test.
Got it, thanks for help confirm this problem.
> 
> Can I have access to your test machine to experiment some solutions?

Sure, I have prepared environment for you, please feel free to ping me if there is anything to do:
Host: 10.72.136.28/kvmautotest
Guest: /root/guest_1.sh & /root/guest_2.sh passwd: kvmautotest

Thanks
Lei

Using this syntax, it works for me:

scp -P 22 root@[fe80::98e1:82ff:fe8d:28b3%switch]:/root/src_ipv6_test root@[fe80::98e1:82ff:fe8d:28b4%switch]:/root/src_ipv6_test

(In reply to Laurent Vivier from comment #33)
> Using this syntax, it works for me:
>

Hi Laurent
 
> scp -P 22 root@[fe80::98e1:82ff:fe8d:28b3%switch]:/root/src_ipv6_test
> root@[fe80::98e1:82ff:fe8d:28b4%switch]:/root/src_ipv6_test

Thanks for help debug this problem, this syntax also can works for me. So from the QE perspective this bug should be moved to a NOTABUG, and QE will based on this syntax to modify test case. Please correct me if I'm wrong.

Regards and Thanks
Lei