Bug 1920408

Summary: Submariner IPsec connections: loaded 9, active 2
Product: Red Hat Advanced Cluster Management for Kubernetes Reporter: Noam Manos <nmanos>
Component: SubmarinerAssignee: Sridhar Gaddam <sgaddam>
Status: CLOSED ERRATA QA Contact: Noam Manos <nmanos>
Severity: high Docs Contact: Christopher Dawson <cdawson>
Priority: unspecified    
Version: rhacm-2.2CC: majopela, nyechiel, sgaddam, smattar, tfreger
Target Milestone: ---Flags: smattar: rhacm-2.2+
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: 0.8.1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-03-04 12:40:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
ipsec pod log
none
Here's how the ipsec connection 9/9 active (bug fixed) none

Description Noam Manos 2021-01-26 09:28:40 UTC 
Created attachment 1750840 [details]
ipsec pod log

Description of problem:
After installing Submariner on AWS and OSP clusters, the Active gateway shows that not all connections were established (2 out of 9 IPSec tunnels)

Version-Release number of selected component (if applicable):
Submariner 0.8.0

How reproducible:
Sometimes

Steps to Reproduce:
https://qe-jenkins-csb-skynet.cloud.paas.psi.redhat.com/job/Submariner-OSP-AWS-No-Overlapping/160/Test-Report

Actual results:

$ subctl show all

Showing information for cluster "pkomarov-cluster-a":
    Discovered network details:
        Network plugin:  OpenShiftSDN
        Service CIDRs:   [172.31.0.0/16]
        Cluster CIDRs:   [10.132.0.0/14]

CLUSTER ID                    ENDPOINT IP     PUBLIC IP       CABLE DRIVER        TYPE            
pkomarov-cluster-a            10.1.64.160     18.225.31.220   libreswan           local           
default-cl2                   10.2.0.206      66.187.232.129  libreswan           remote          

GATEWAY                         CLUSTER                 REMOTE IP       CABLE DRIVER        SUBNETS                                 STATUS          
default-cl2-mr8gk-worker-kskb2  default-cl2             10.2.0.206      libreswan           172.32.0.0/16, 10.136.0.0/14            connected       

NODE                            HA STATUS       SUMMARY                         
ip-10-1-64-160                  active          All connections (1) are established

COMPONENT                       REPOSITORY                                            VERSION         
submariner                      registry.redhat.io/rhacm2-tech-preview                v0.8.0          
submariner-operator             registry.redhat.io/rhacm2-tech-preview/submariner-rhe v0.8.0          
service-discovery               registry.redhat.io/rhacm2-tech-preview                v0.8.0          

Showing information for cluster "default-cl2":
    Discovered network details:
        Network plugin:  OpenShiftSDN
        Service CIDRs:   [172.32.0.0/16]
        Cluster CIDRs:   [10.136.0.0/14]

CLUSTER ID                    ENDPOINT IP     PUBLIC IP       CABLE DRIVER        TYPE            
default-cl2                   10.2.0.206      66.187.232.129  libreswan           local           
pkomarov-cluster-a            10.1.64.160     18.225.31.220   libreswan           remote          

GATEWAY                         CLUSTER                 REMOTE IP       CABLE DRIVER        SUBNETS                                 STATUS          
ip-10-1-64-160                  pkomarov-cluster-a      10.1.64.160     libreswan           172.31.0.0/16, 10.132.0.0/14            connected       

NODE                            HA STATUS       SUMMARY                         
default-cl2-mr8gk-worker-kskb2  active          All connections (1) are established

COMPONENT                       REPOSITORY                                            VERSION         
submariner                      registry.redhat.io/rhacm2-tech-preview                v0.8.0          
submariner-operator             registry.redhat.io/rhacm2-tech-preview/submariner-rhe v0.8.0          
service-discovery               registry.redhat.io/rhacm2-tech-preview                v0.8.0  

# However, looking at the active gateway, I see that not all IPSec connections were established:

000 Total IPsec connections: loaded 9, active 2
000  
000 State Information: DDoS cookies not required, Accepting new IKE connections
000 IKE SAs: total(2), half-open(1), open(0), authenticated(1), anonymous(0)
000 IPsec SAs: total(2), authenticated(2), anonymous(0)
000  
000 #5: "submariner-cable-default-cl2-10-2-0-206-0-0":53058 STATE_V2_ESTABLISHED_CHILD_SA (IPsec SA established); EVENT_SA_REKEY in 28518s; newest IPSEC; eroute owner; isakmp#4; idle;
000 #5: "submariner-cable-default-cl2-10-2-0-206-0-0" esp.eda0e601.232.129 esp.7f2df5b4.64.160 tun.0.232.129 tun.0.64.160 Traffic: ESPin=0B ESPout=0B! ESPmax=0B 
000 #7: "submariner-cable-default-cl2-10-2-0-206-0-0":4501 STATE_PARENT_I1 (sent IKE_SA_INIT request); EVENT_RETRANSMIT in 0s; idle;
000 #7: pending CHILD SA for "submariner-cable-default-cl2-10-2-0-206-0-0"
000 #7: pending CHILD SA for "submariner-cable-default-cl2-10-2-0-206-0-0"
000 #7: pending CHILD SA for "submariner-cable-default-cl2-10-2-0-206-0-0"
000 #7: pending CHILD SA for "submariner-cable-default-cl2-10-2-0-206-0-0"
000 #7: pending CHILD SA for "submariner-cable-default-cl2-10-2-0-206-0-0"
000 #7: pending CHILD SA for "submariner-cable-default-cl2-10-2-0-206-0-0"
000 #7: pending CHILD SA for "submariner-cable-default-cl2-10-2-0-206-0-0"
000 #7: pending CHILD SA for "submariner-cable-default-cl2-10-2-0-206-0-0"
000 #7: pending CHILD SA for "submariner-cable-default-cl2-10-2-0-206-0-0"
000 #4: "submariner-cable-default-cl2-10-2-0-206-2-2":53058 STATE_V2_ESTABLISHED_IKE_SA (established IKE SA); EVENT_SA_REKEY in 3318s; newest ISAKMP; idle;
000 #6: "submariner-cable-default-cl2-10-2-0-206-2-2":53058 STATE_V2_ESTABLISHED_CHILD_SA (IPsec SA established); EVENT_SA_REKEY in 28519s; newest IPSEC; eroute owner; isakmp#4; idle;
000 #6: "submariner-cable-default-cl2-10-2-0-206-2-2" esp.16339c47.232.129 esp.5c3520a.64.160 tun.0.232.129 tun.0.64.160 Traffic: ESPin=924B ESPout=924B! ESPmax=0B 


Expected results:
The number of loaded ipsec connections, should be the same number of the active ones.

Additional info:
Full gateway pod log attached
Comment 2 Mike Ng 2021-01-29 14:32:44 UTC 
G2Bsync 768992593 comment 
 nyechiel Thu, 28 Jan 2021 11:33:29 UTC 
 G2Bsync This seems like a Libreswan/IPsec issues which is being investigated here: https://github.com/submariner-io/submariner/issues/1081
Comment 4 Noam Manos 2021-02-03 15:01:28 UTC 
Created attachment 1754799 [details]
Here's how the ipsec connection 9/9 active (bug fixed)
Comment 7 errata-xmlrpc 2021-03-04 12:40:34 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (RHEA: Submariner 0.8 - bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2021:0728