Bug 2326136 - trafficserver stopped working after upgrade from F39 to F40 - connection to backendserver is finished before data flows
Summary: trafficserver stopped working after upgrade from F39 to F40 - connection to b...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: trafficserver
Version: 40
Hardware: Unspecified
OS: Linux
unspecified
urgent
Target Milestone: ---
Assignee: Jered Floyd
QA Contact: Fedora Extras Quality Assurance
URL: https://github.com/apache/trafficserv...
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2024-11-13 23:28 UTC by customercare
Modified: 2024-11-22 03:21 UTC (History)
1 user (show)

Fixed In Version: trafficserver-9.2.6-2.fc39 trafficserver-9.2.6-2.el9 trafficserver-9.2.6-2.el8 trafficserver-9.2.6-2.fc40 trafficserver-9.2.6-2.fc41
Clone Of:
Environment:
Last Closed: 2024-11-22 02:09:47 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)

Description customercare 2024-11-13 23:28:06 UTC
OS: Fedora 40
Version: 9.2.5

Worked on Fedora 39 with same version, downgrade to fc39 version same release tested, same result.

It does not matter where the backendserver is, local on another port or on a different ip.
This dump is locally on the same server:

22:48:01.224680 IP 83.246.80.131.53420 > 83.246.80.131.18443: Flags [S], seq 4065112379, win 65495, options [mss 65495,sackOK,TS val 2381864768 ecr 0,nop,wscale 7], length 0
22:48:01.224689 IP 83.246.80.131.18443 > 83.246.80.131.53420: Flags [S.], seq 2155888146, ack 4065112380, win 65483, options [mss 65495,sackOK,TS val 2381864768 ecr 2381864768,nop,wscale 7], length 0
22:48:01.224698 IP 83.246.80.131.53420 > 83.246.80.131.18443: Flags [.], ack 1, win 512, options [nop,nop,TS val 2381864768 ecr 2381864768], length 0
22:48:01.224994 IP 83.246.80.131.53420 > 83.246.80.131.18443: Flags [F.], seq 1, ack 1, win 512, options [nop,nop,TS val 2381864769 ecr 2381864768], length 0
22:48:01.225168 IP 83.246.80.131.18443 > 83.246.80.131.53420: Flags [F.], seq 1, ack 2, win 512, options [nop,nop,TS val 2381864769 ecr 2381864769], length 0
22:48:01.225178 IP 83.246.80.131.53420 > 83.246.80.131.18443: Flags [.], ack 2, win 512, options [nop,nop,TS val 2381864769 ecr 2381864769], length 0

Immediatly after the SYN-ACK is ACKed ATS sends a FIN package. And than it logs:

20241113.22h48m01s CONNECT: attempt fail [CONNECTION_ERROR] to 83.246.80.131:18443 for host='cloud-foo.de' connection_result=Invalid argument [22] error=Invalid argument [22] attempts=3 url='https://cloud-foo.de:18443/equery.js'

A connect with curl to that ip:port works flawless.. The cert is the same as ATS would use.

For more infos see : 

https://github.com/apache/trafficserver/issues/11858

Reproducible: Always

Steps to Reproduce:
Tcpdump the connection to a backendserver 
call a webpage that is not cached
Actual Results:  
The outgoing connection, no matter it local or remote desto, just does this

C: SYN
S: SYN-ACK
C: ACK
C: FIN
S: FIN
C: ACK

Expected Results:  
A working connection to a given desto

- no firewall is in the way
- downgrades of:
 - HTTP ( backendserver ) 
 - TrafficServer 
 - SystemD ( Processlimits ) 
 tested -> same result
- same kernel as before the update -> same result
- connecting to backendserver with curl/firefox .. works
- crypto policy downgraded -> same result
- mod_ssl downgrade -> same result
- diag debugging enabled -> no clues why it does not send data after SYN-ACK/ACK
- firewall cleared
- nftables not in use
- firewalld not in use
- SELinux checked - disabled -> no effect

I'm out of options

Comment 1 Fedora Update System 2024-11-13 23:37:08 UTC
FEDORA-EPEL-2024-60dd7e7ad3 (trafficserver-9.2.6-2.el8) has been submitted as an update to Fedora EPEL 8.
https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2024-60dd7e7ad3

Comment 2 Fedora Update System 2024-11-13 23:37:08 UTC
FEDORA-2024-f4dc07db08 (trafficserver-9.2.6-2.fc41) has been submitted as an update to Fedora 41.
https://bodhi.fedoraproject.org/updates/FEDORA-2024-f4dc07db08

Comment 3 Fedora Update System 2024-11-13 23:37:09 UTC
FEDORA-2024-b3c4e8da81 (trafficserver-9.2.6-2.fc40) has been submitted as an update to Fedora 40.
https://bodhi.fedoraproject.org/updates/FEDORA-2024-b3c4e8da81

Comment 4 Fedora Update System 2024-11-13 23:37:09 UTC
FEDORA-2024-589ea34c42 (trafficserver-9.2.6-2.fc39) has been submitted as an update to Fedora 39.
https://bodhi.fedoraproject.org/updates/FEDORA-2024-589ea34c42

Comment 5 Jered Floyd 2024-11-13 23:46:57 UTC
Great timing on your report -- I hit this on the RHEL 9.5 update last night, where this also occurs.  This is due to https://github.com/apache/trafficserver/issues/11465  where outbound TLS connections cannot be established to origin servers because of an incompatibility with OpenSSL 3.2+.  Fedora 40 is the first Fedora release that uses 3.2+, and the RHEL 9.5 update also moves to OpenSSL 3.2.2 (which led to a bit of a panic for me last night).

Appreciated if you could try out the testing build and verify the fix in your environment (and vote in Bodhi so it goes to release faster).

Comment 6 customercare 2024-11-14 00:09:04 UTC
done and karma given.. works again.

Comment 7 Fedora Update System 2024-11-14 02:14:18 UTC
FEDORA-2024-f4dc07db08 has been pushed to the Fedora 41 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2024-f4dc07db08`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2024-f4dc07db08

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 8 Fedora Update System 2024-11-14 03:08:37 UTC
FEDORA-EPEL-2024-23b122f235 has been pushed to the Fedora EPEL 9 testing repository.

You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2024-23b122f235

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 9 Fedora Update System 2024-11-14 03:15:05 UTC
FEDORA-EPEL-2024-60dd7e7ad3 has been pushed to the Fedora EPEL 8 testing repository.

You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2024-60dd7e7ad3

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 10 Fedora Update System 2024-11-14 03:32:30 UTC
FEDORA-2024-b3c4e8da81 has been pushed to the Fedora 40 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2024-b3c4e8da81`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2024-b3c4e8da81

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 11 Fedora Update System 2024-11-14 06:06:05 UTC
FEDORA-2024-589ea34c42 has been pushed to the Fedora 39 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2024-589ea34c42`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2024-589ea34c42

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 12 Fedora Update System 2024-11-22 02:09:47 UTC
FEDORA-2024-589ea34c42 (trafficserver-9.2.6-2.fc39) has been pushed to the Fedora 39 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 13 Fedora Update System 2024-11-22 02:39:47 UTC
FEDORA-EPEL-2024-23b122f235 (trafficserver-9.2.6-2.el9) has been pushed to the Fedora EPEL 9 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 14 Fedora Update System 2024-11-22 02:55:14 UTC
FEDORA-EPEL-2024-60dd7e7ad3 (trafficserver-9.2.6-2.el8) has been pushed to the Fedora EPEL 8 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 15 Fedora Update System 2024-11-22 03:21:13 UTC
FEDORA-2024-b3c4e8da81 (trafficserver-9.2.6-2.fc40) has been pushed to the Fedora 40 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 16 Fedora Update System 2024-11-22 03:21:20 UTC
FEDORA-2024-f4dc07db08 (trafficserver-9.2.6-2.fc41) has been pushed to the Fedora 41 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.