Bug 2092234

Summary: subscription-manager creates new TCP and TLS connection for each HTTP request
Product: Red Hat Enterprise Linux 9 Reporter: Jiri Hnidek <jhnidek>
Component: subscription-managerAssignee: Jiri Hnidek <jhnidek>
Status: CLOSED ERRATA QA Contact: Red Hat subscription-manager QE Team <rhsm-qe>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 9.0CC: arpandey, jsefler, mhorky, redakkan, zpetrace
Target Milestone: rcKeywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-11-15 11:19:33 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jiri Hnidek 2022-06-01 07:27:04 UTC 
Description of problem:
All tools provided by subscription-manager RPM and all tools using Python package rhsm from python3-subscription-manager-rhsm create new TCP and TLS connection for each HTTP REST API call. This is not efficient, because each TLS handshake requires sending 3 packets and TLS handshake requires sending 4 packets. Shutdown of TCP/TLS also requires sending extra packets. When the TTL of connection between client and server is about 100ms (typical for clients in Europe and Asia), then it can lead to significant delays.

Version-Release number of selected component (if applicable):

[root@localhost ~]# subscription-manager version
server type: This system is currently not registered.
subscription management server: 4.2.1-1
subscription management rules: 5.43
subscription-manager: 1.29.26

How reproducible:
100%

Steps to Reproduce:
1. Open Wireshark in privileged mode:
   [user@localhost ~]$ sudo wireshark

2. Start capturing traffic between client and server with appropriate filter to see only communication between sub-man and candlepin server.
   Capture -> Options, Select interface "any", Set capture filter: e.g. "tcp port 443 and host 10.2.89.172" (10.2.89.172 is IP address of testing candlepin server)

3. Run any subscription-manager sub-command e.g. "subscription-manager version"
4. Stop capturing of traffic

Actual results:
Each REST API call opens own TCP/TLS connection. You can see many TCP and TLS handshake. There is also not proper TLS teardown (shutdown handhake) at the end of TLS connection.

Expected results:
Only one TCP and TLS handshake. TLS connection is finished gracefully.

Additional info:
Comment 1 Zdenek Petracek 2022-06-30 11:20:22 UTC 
Reproducing the bug:
Version:
[root@kvm-02-guest05 ~]# subscription-manager version
server type: This system is currently not registered.
subscription management server: 4.0.18-1
subscription management rules: 5.41
subscription-manager: 1.29.26-4.el9

Capturing the traffic after running subscription-manager version:
[root@kvm-02-guest05 ~]# tshark -i any host 10.2.77.208
Running as user "root" and group "root". This could be dangerous.
Capturing on 'any'
    1 0.000000000 10.37.153.79 → 10.2.77.208  TCP 76 40364 → 443 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=2120740349 TSecr=0 WS=128
    2 0.104841239  10.2.77.208 → 10.37.153.79 TCP 76 443 → 40364 [SYN, ACK] Seq=0 Ack=1 Win=26844 Len=0 MSS=1387 SACK_PERM=1 TSval=1738666390 TSecr=2120740349 WS=16
    3 0.104892470 10.37.153.79 → 10.2.77.208  TCP 68 40364 → 443 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=2120740454 TSecr=1738666390
    4 0.106091178 10.37.153.79 → 10.2.77.208  TLSv1 374 Client Hello
    5 0.210649526  10.2.77.208 → 10.37.153.79 TCP 68 443 → 40364 [ACK] Seq=1 Ack=307 Win=27920 Len=0 TSval=1738666496 TSecr=2120740456
    6 0.215742666  10.2.77.208 → 10.37.153.79 TLSv1.2 5018 Server Hello, Certificate, Server Key Exchange, Certificate Request, Server Hello Done
    7 0.215784404 10.37.153.79 → 10.2.77.208  TCP 68 40364 → 443 [ACK] Seq=307 Ack=4951 Win=61568 Len=0 TSval=2120740565 TSecr=1738666501
    8 0.221755825 10.37.153.79 → 10.2.77.208  TLSv1.2 173 Certificate, Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message
    9 0.326816041  10.2.77.208 → 10.37.153.79 TLSv1.2 358 New Session Ticket, Change Cipher Spec, Encrypted Handshake Message
   10 0.326836569 10.37.153.79 → 10.2.77.208  TCP 68 40364 → 443 [ACK] Seq=412 Ack=5241 Win=64128 Len=0 TSval=2120740676 TSecr=1738666612
   11 0.327268647 10.37.153.79 → 10.2.77.208  TLSv1.2 481 Application Data
   12 0.443443307  10.2.77.208 → 10.37.153.79 TLSv1.2 1431 Application Data
   13 0.443515625 10.37.153.79 → 10.2.77.208  TCP 68 40364 → 443 [ACK] Seq=825 Ack=6604 Win=64128 Len=0 TSval=2120740793 TSecr=1738666729
   14 0.447268108 10.37.153.79 → 10.2.77.208  TCP 68 40364 → 443 [FIN, ACK] Seq=825 Ack=6604 Win=64128 Len=0 TSval=2120740797 TSecr=1738666729
   15 0.479194295 10.37.153.79 → 10.2.77.208  TCP 76 40366 → 443 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=2120740829 TSecr=0 WS=128
   16 0.552238492  10.2.77.208 → 10.37.153.79 TCP 68 443 → 40364 [FIN, ACK] Seq=6604 Ack=826 Win=28992 Len=0 TSval=1738666838 TSecr=2120740797
   17 0.552316353 10.37.153.79 → 10.2.77.208  TCP 68 40364 → 443 [ACK] Seq=826 Ack=6605 Win=64128 Len=0 TSval=2120740902 TSecr=1738666838
   18 0.583796243  10.2.77.208 → 10.37.153.79 TCP 76 443 → 40366 [SYN, ACK] Seq=0 Ack=1 Win=26844 Len=0 MSS=1387 SACK_PERM=1 TSval=1738666869 TSecr=2120740829 WS=16
   19 0.583824129 10.37.153.79 → 10.2.77.208  TCP 68 40366 → 443 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=2120740933 TSecr=1738666869
   20 0.584414182 10.37.153.79 → 10.2.77.208  TLSv1 374 Client Hello
   21 0.689038587  10.2.77.208 → 10.37.153.79 TCP 68 443 → 40366 [ACK] Seq=1 Ack=307 Win=27920 Len=0 TSval=1738666975 TSecr=2120740934
   22 0.695735740  10.2.77.208 → 10.37.153.79 TLSv1.2 5018 Server Hello, Certificate, Server Key Exchange, Certificate Request, Server Hello Done
   23 0.695769273 10.37.153.79 → 10.2.77.208  TCP 68 40366 → 443 [ACK] Seq=307 Ack=4951 Win=61568 Len=0 TSval=2120741045 TSecr=1738666981
   24 0.700520673 10.37.153.79 → 10.2.77.208  TLSv1.2 173 Certificate, Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message
   25 0.807437415  10.2.77.208 → 10.37.153.79 TLSv1.2 358 New Session Ticket, Change Cipher Spec, Encrypted Handshake Message
   26 0.807511589 10.37.153.79 → 10.2.77.208  TCP 68 40366 → 443 [ACK] Seq=412 Ack=5241 Win=64128 Len=0 TSval=2120741157 TSecr=1738667093
   27 0.808020817 10.37.153.79 → 10.2.77.208  TLSv1.2 487 Application Dataserver type: This system is currently not registered.
   28 0.926882698  10.2.77.208 → 10.37.153.79 TLSv1.2 1006 Application Data
   29 0.926911889 10.37.153.79 → 10.2.77.208  TCP 68 40366 → 443 [ACK] Seq=831 Ack=6179 Win=64128 Len=0 TSval=2120741276 TSecr=1738667212
   30 0.929591620 10.37.153.79 → 10.2.77.208  TCP 68 40366 → 443 [FIN, ACK] Seq=831 Ack=6179 Win=64128 Len=0 TSval=2120741279 TSecr=1738667212
   31 1.034915631  10.2.77.208 → 10.37.153.79 TCP 68 443 → 40366 [FIN, ACK] Seq=6179 Ack=832 Win=28992 Len=0 TSval=1738667320 TSecr=2120741279
   32 1.034944736 10.37.153.79 → 10.2.77.208  TCP 68 40366 → 443 [ACK] Seq=832 Ack=6180 Win=64128 Len=0 TSval=2120741384 TSecr=1738667320
^C32 packets captured

^^ you can see many TCP/TLS handshakes there 

Pre-verifying:
Version:
[root@kvm-02-guest05 ~]# subscription-manager version
server type: This system is currently not registered.
subscription management server: 4.0.18-1
subscription management rules: 5.41
subscription-manager: 1.29.28+29.gb8fc6ecfb-1.git.0.0cce761

Capturing the traffic after running subscription-manager version:
[root@kvm-02-guest05 ~]# tshark -i any host 10.2.77.208
Running as user "root" and group "root". This could be dangerous.
Capturing on 'any'
    1 0.000000000 10.37.153.79 → 10.2.77.208  TCP 76 46062 → 443 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=2119049798 TSecr=0 WS=128
    2 0.104908506  10.2.77.208 → 10.37.153.79 TCP 76 443 → 46062 [SYN, ACK] Seq=0 Ack=1 Win=26844 Len=0 MSS=1387 SACK_PERM=1 TSval=1736976008 TSecr=2119049798 WS=16
    3 0.104962799 10.37.153.79 → 10.2.77.208  TCP 68 46062 → 443 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=2119049903 TSecr=1736976008
    4 0.108086145 10.37.153.79 → 10.2.77.208  TLSv1 374 Client Hello
    5 0.212759678  10.2.77.208 → 10.37.153.79 TCP 68 443 → 46062 [ACK] Seq=1 Ack=307 Win=27920 Len=0 TSval=1736976116 TSecr=2119049906
    6 0.218053867  10.2.77.208 → 10.37.153.79 TLSv1.2 5018 Server Hello, Certificate, Server Key Exchange, Certificate Request, Server Hello Done
    7 0.218081834 10.37.153.79 → 10.2.77.208  TCP 68 46062 → 443 [ACK] Seq=307 Ack=4951 Win=61568 Len=0 TSval=2119050016 TSecr=1736976121
    8 0.225152595 10.37.153.79 → 10.2.77.208  TLSv1.2 173 Certificate, Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message
    9 0.330295440  10.2.77.208 → 10.37.153.79 TLSv1.2 358 New Session Ticket, Change Cipher Spec, Encrypted Handshake Message
   10 0.330317283 10.37.153.79 → 10.2.77.208  TCP 68 46062 → 443 [ACK] Seq=412 Ack=5241 Win=64128 Len=0 TSval=2119050128 TSecr=1736976233
   11 0.332973960 10.37.153.79 → 10.2.77.208  TLSv1.2 529 Application Data
   12 0.447565283  10.2.77.208 → 10.37.153.79 TLSv1.2 1431 Application Data
   13 0.447637701 10.37.153.79 → 10.2.77.208  TCP 68 46062 → 443 [ACK] Seq=873 Ack=6604 Win=64128 Len=0 TSval=2119050246 TSecr=1736976350
   14 0.462741354 10.37.153.79 → 10.2.77.208  TLSv1.2 559 Application Data
   15 0.583425153  10.2.77.208 → 10.37.153.79 TLSv1.2 1006 Application Data
   16 0.589665016 10.37.153.79 → 10.2.77.208  TLSv1.2 99 Encrypted Alert
   17 0.694972210  10.2.77.208 → 10.37.153.79 TCP 68 443 → 46062 [FIN, ACK] Seq=7542 Ack=1395 Win=30064 Len=0 TSval=1736976598 TSecr=2119050388
   18 0.695335552 10.37.153.79 → 10.2.77.208  TCP 68 46062 → 443 [FIN, ACK] Seq=1395 Ack=7543 Win=64128 Len=0 TSval=2119050493 TSecr=1736976598
   19 0.799897702  10.2.77.208 → 10.37.153.79 TCP 68 443 → 46062 [ACK] Seq=7543 Ack=1396 Win=30064 Len=0 TSval=1736976703 TSecr=2119050493
^C19 packets captured

^^much less traffic, TLS connection is finished -- PASSED
Comment 4 Archana Pandey 2022-07-27 09:26:57 UTC 
Verifying on nightly compose of RHEL 9.1 with subscription-manager-1.29.29-1.el9.x86_64

 Beaker Test information:
                         HOSTNAME=kvm-03-guest09.hv2.lab.eng.bos.redhat.com
                            JOBID=6853835
                         RECIPEID=12345831
                    RESULT_SERVER=
                           DISTRO=RHEL-9.1.0-20220721.1
                     ARCHITECTURE=x86_64

[root@kvm-03-guest09 ~]# subscription-manager version
server type: Red Hat Subscription Management
subscription management server: 4.0.18-2
subscription management rules: 5.41
subscription-manager: 1.29.29-1.el9

>> captured traffic while running above command- 
[root@kvm-03-guest09 ~]# tshark -i ens3 host 10.2.77.208 
Running as user "root" and group "root". This could be dangerous.
Capturing on 'ens3'
    1 0.000000000  10.16.56.44 → 10.2.77.208  TCP 74 49238 → 443 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=1765216341 TSecr=0 WS=128
    2 0.012596600  10.2.77.208 → 10.16.56.44  TCP 74 443 → 49238 [SYN, ACK] Seq=0 Ack=1 Win=26844 Len=0 MSS=8960 SACK_PERM=1 TSval=2909427071 TSecr=1765216341 WS=16
    3 0.012615128  10.16.56.44 → 10.2.77.208  TCP 66 49238 → 443 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=1765216353 TSecr=2909427071
    4 0.012995372  10.16.56.44 → 10.2.77.208  TLSv1 372 Client Hello
    5 0.025442365  10.2.77.208 → 10.16.56.44  TCP 66 443 → 49238 [ACK] Seq=1 Ack=307 Win=27920 Len=0 TSval=2909427084 TSecr=1765216354
    6 0.030662321  10.2.77.208 → 10.16.56.44  TLSv1.2 5016 Server Hello, Certificate, Server Key Exchange, Certificate Request, Server Hello Done
    7 0.030699862  10.16.56.44 → 10.2.77.208  TCP 66 49238 → 443 [ACK] Seq=307 Ack=4951 Win=62080 Len=0 TSval=1765216371 TSecr=2909427089
    8 0.039966252  10.16.56.44 → 10.2.77.208  TLSv1.2 2364 Certificate, Client Key Exchange, Certificate Verify, Change Cipher Spec, Encrypted Handshake Message
    9 0.052633958  10.2.77.208 → 10.16.56.44  TCP 66 443 → 49238 [ACK] Seq=4951 Ack=2605 Win=45824 Len=0 TSval=2909427111 TSecr=1765216381
   10 0.053591532  10.2.77.208 → 10.16.56.44  TLSv1.2 2020 New Session Ticket, Change Cipher Spec, Encrypted Handshake Message
   11 0.053607953  10.16.56.44 → 10.2.77.208  TCP 66 49238 → 443 [ACK] Seq=2605 Ack=6905 Win=63616 Len=0 TSval=1765216394 TSecr=2909427112
   12 0.054141171  10.16.56.44 → 10.2.77.208  TLSv1.2 479 Application Data
   13 0.081789644  10.2.77.208 → 10.16.56.44  TLSv1.2 1429 Application Data
   14 0.081835874  10.16.56.44 → 10.2.77.208  TCP 66 49238 → 443 [ACK] Seq=3018 Ack=8268 Win=64128 Len=0 TSval=1765216422 TSecr=2909427140
   15 0.153946957  10.16.56.44 → 10.2.77.208  TLSv1.2 509 Application Data
   16 0.185931076  10.2.77.208 → 10.16.56.44  TLSv1.2 1004 Application Data
   17 0.185971323  10.16.56.44 → 10.2.77.208  TCP 66 49238 → 443 [ACK] Seq=3461 Ack=9206 Win=64128 Len=0 TSval=1765216526 TSecr=2909427244
   18 0.300138711  10.16.56.44 → 10.2.77.208  TLSv1.2 97 Encrypted Alert
   19 0.313120870  10.2.77.208 → 10.16.56.44  TCP 66 443 → 49238 [FIN, ACK] Seq=9206 Ack=3492 Win=51616 Len=0 TSval=2909427371 TSecr=1765216641
   20 0.313257894  10.16.56.44 → 10.2.77.208  TCP 66 49238 → 443 [FIN, ACK] Seq=3492 Ack=9207 Win=64128 Len=0 TSval=1765216654 TSecr=2909427371
   21 0.325886442  10.2.77.208 → 10.16.56.44  TCP 66 443 → 49238 [ACK] Seq=9207 Ack=3493 Win=51616 Len=0 TSval=2909427384 TSecr=1765216654


trying another request-
[root@kvm-03-guest09 ~]# subscription-manager identity
system identity: dbd7fe04-7527-476e-b62f-7552447e3763
name: kvm-03-guest09.hv2.lab.eng.bos.redhat.com
org name: 13343512
org ID: 13343512

>> captured Traffic - 
22 14.985452821  10.16.56.44 → 10.2.77.208  TCP 74 42042 → 443 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=1765231326 TSecr=0 WS=128
   23 14.998045769  10.2.77.208 → 10.16.56.44  TCP 74 443 → 42042 [SYN, ACK] Seq=0 Ack=1 Win=26844 Len=0 MSS=8960 SACK_PERM=1 TSval=2909442055 TSecr=1765231326 WS=16
   24 14.998096790  10.16.56.44 → 10.2.77.208  TCP 66 42042 → 443 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=1765231339 TSecr=2909442055
   25 14.998567573  10.16.56.44 → 10.2.77.208  TLSv1 372 Client Hello
   26 15.011110175  10.2.77.208 → 10.16.56.44  TCP 66 443 → 42042 [ACK] Seq=1 Ack=307 Win=27920 Len=0 TSval=2909442068 TSecr=1765231339
   27 15.016097001  10.2.77.208 → 10.16.56.44  TLSv1.2 4410 Server Hello, Certificate, Server Key Exchange
   28 15.016097175  10.2.77.208 → 10.16.56.44  TLSv1.2 672 Certificate Request, Server Hello Done
   29 15.016112580  10.16.56.44 → 10.2.77.208  TCP 66 42042 → 443 [ACK] Seq=307 Ack=4345 Win=62848 Len=0 TSval=1765231357 TSecr=2909442073
   30 15.016124775  10.16.56.44 → 10.2.77.208  TCP 66 42042 → 443 [ACK] Seq=307 Ack=4951 Win=62336 Len=0 TSval=1765231357 TSecr=2909442073
   31 15.025156801  10.16.56.44 → 10.2.77.208  TLSv1.2 2364 Certificate, Client Key Exchange, Certificate Verify, Change Cipher Spec, Encrypted Handshake Message
   32 15.037673411  10.2.77.208 → 10.16.56.44  TCP 66 443 → 42042 [ACK] Seq=4951 Ack=2605 Win=48720 Len=0 TSval=2909442094 TSecr=1765231366
   33 15.042962025  10.2.77.208 → 10.16.56.44  TLSv1.2 2020 New Session Ticket, Change Cipher Spec, Encrypted Handshake Message
   34 15.042985992  10.16.56.44 → 10.2.77.208  TCP 66 42042 → 443 [ACK] Seq=2605 Ack=6905 Win=63616 Len=0 TSval=1765231384 TSecr=2909442100
   35 15.043369754  10.16.56.44 → 10.2.77.208  TLSv1.2 531 Application Data
   36 15.096732135  10.2.77.208 → 10.16.56.44  TCP 66 443 → 42042 [ACK] Seq=6905 Ack=3070 Win=51616 Len=0 TSval=2909442154 TSecr=1765231384
   37 15.100405380  10.2.77.208 → 10.16.56.44  TLSv1.2 860 Application Data
   38 15.100429718  10.16.56.44 → 10.2.77.208  TCP 66 42042 → 443 [ACK] Seq=3070 Ack=7699 Win=64128 Len=0 TSval=1765231441 TSecr=2909442157
   39 15.130774127  10.16.56.44 → 10.2.77.208  TLSv1.2 97 Encrypted Alert
   40 15.143382481  10.2.77.208 → 10.16.56.44  TCP 66 443 → 42042 [ACK] Seq=7699 Ack=3101 Win=51616 Len=0 TSval=2909442200 TSecr=1765231471
   41 15.143382977  10.2.77.208 → 10.16.56.44  TCP 66 443 → 42042 [FIN, ACK] Seq=7699 Ack=3101 Win=51616 Len=0 TSval=2909442200 TSecr=1765231471
   42 15.143624797  10.16.56.44 → 10.2.77.208  TCP 66 42042 → 443 [FIN, ACK] Seq=3101 Ack=7700 Win=64128 Len=0 TSval=1765231484 TSecr=2909442200
   43 15.156064857  10.2.77.208 → 10.16.56.44  TCP 66 443 → 42042 [ACK] Seq=7700 Ack=3102 Win=51616 Len=0 TSval=2909442213 TSecr=1765231484


Verified that traffic has reduces and the TLS connections are getting closed properly

PASSED
Comment 6 errata-xmlrpc 2022-11-15 11:19:33 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (subscription-manager bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:8341