Bug 1414592 - Web logins fail behind AWS load balancer intermittently
Summary: Web logins fail behind AWS load balancer intermittently
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: UI - OPS
Version: 5.7.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: GA
: cfme-future
Assignee: Martin Povolny
QA Contact: Mike Shriver
URL:
Whiteboard: load_balancer:ec2:ui
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-01-18 22:41 UTC by Jared Deubel
Modified: 2019-07-16 23:32 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Category: Bug
Cloudforms Team: CFME Core
Target Upstream Version:


Attachments (Terms of Use)
production.log (118.16 KB, text/plain)
2017-01-18 22:42 UTC, Jared Deubel
no flags Details

Description Jared Deubel 2017-01-18 22:41:29 UTC
Description of problem:
Customer has two UI appliances running within AWS, behind an AWS load balancer. These can consistently be accessed directly, with no issue.  Between client and myself, the LB connection occasionally denies authentication (or other web access) past the login screen.

The logs show CSRF token authentication validation errors.

Allegedly, the LB is configured with sticky sessions, and regardless, both the appliances have the advanced setting "session_store: sql"

Where are you experiencing the behavior?  What environment?

Brand new AWS deployment

Version-Release number of selected component (if applicable):
5.7

Comment 2 Jared Deubel 2017-01-18 22:42:49 UTC
Created attachment 1242295 [details]
production.log

Comment 4 Dave Johnson 2017-01-23 15:43:04 UTC
Matous, can you try to reproduce this please.  Maybe with Azure or GCE LBs as well.

Comment 5 Matt Pusateri 2017-01-24 15:57:06 UTC
1. Do we have any idea of how their network in AWS is laid out? 
2. Are they using multiple private subnets in a VPC?  
3. Do they have any specific Network ACL's or Security groups?
4. What are they using internally for DNS and does the ELB point at hostnames or IP's?

I think we need more info to duplicate the environment.

Comment 6 Matouš Mojžíš 2017-01-24 16:05:54 UTC
2. There are multiple subnets for VPC, mostly used for high availability.
3. Yes, both of them. ACLs for subnets, Security groups for the rest.
4. When adding ELB they create endpoint with A record. We need to add CNAME record pointing to that record on our domain then.
When I try to create HTTPS ELB they require primary key and cert file.
I tried to evade it by adding TCP rule on port 443, but it ended up with SSL_ERROR_RX_RECORD_TOO_LONG because appliances use IP address and then I tried to access it from endpoint.
Also when not using http/https ELB I can't apply sticky sessions.
I can try to reproduce it with HTTP only, but it probably can have some impact?

Comment 7 Martin Povolny 2017-02-22 13:04:03 UTC
I setup an experiment with a Fedora VM doing a HTTPS proxy for an appliance.

I immediately run into a problem with absolute URLs being generated for javascript-based redirects.

I have a PR for that:

https://github.com/ManageIQ/manageiq-ui-classic/pull/448

I am looking for other problems that would prevent or complicate the use of CFME behing a proxy or load balancer.

I can see that the 'content-security-policy' containts a hard-coded IP of the apliance:

default-src 'self'; connect-src 'self' wss://192.168.122.187; frame-src 'self'; script-src 'unsafe-eval' 'unsafe-inline' 'self'; style-src 'unsafe-inline' 'self'; report-uri /dashboard/csp_report

that will cause issues with HTML5 base VNC/SPICE consoles and notification drawer, but it can be probably handled using header-editing capabilities of Apache proxy.

My proxy config follows:

  SSLProxyEngine on
  SSLProxyVerify none
  SSLProxyCheckPeerCN off
  SSLProxyCheckPeerName off
  SSLProxyCheckPeerExpire off


  <Proxy balancer://cloudforms>
      BalancerMember https://193.168.122.187:443 ping=60 timeout=300 disablereuse=On route=cformsapp0p
      # BalancerMember http://192.168.122.1:3000 ping=60 timeout=300 disablereuse=On route=cformsapp0p
  </Proxy>

   RequestHeader set X_FORWARDED_PROTO 'https'
   RequestHeader set X-Forwarded-Ssl on
   AllowEncodedSlashes NoDecode

   ProxyPass / balancer://cloudforms/ nocanon
   ProxyPassReverse / balancer://cloudforms/


Anyway we should carry on trying to reproduce the issue on Amazon.

Matous: regarding "When I try to create HTTPS ELB they require primary key and cert file.": you can generate a primary key and certificate using openssl command line.

Comment 8 Martin Povolny 2017-04-18 13:35:25 UTC
The 2 problems we have identified are both fixed: absolute URLs and CSP headers.

I am currently testing the proxying with Apache and HAproxy.

I am putting some notes here: https://github.com/martinpovolny/manageiq-utils/wiki/HA-proxying-ManageIQ-UI-Classic (until I am done and will write an article about this).

I can use some help testing on Amazon.

Comment 13 Josh Carter 2018-10-02 14:32:43 UTC
Dear customer, 

The CloudForms team is reviewing the current CloudForms Bug(defect) backlog in order to target engineering efforts. We are closing any bugs for versions that no longer have an active errata stream or that have hit their age limit. We are committing to better management of the backlog as we move forward. If you have an bug that you are still able to reproduce on a current version of CloudForms please open a new bug. 

If you have any concerns about this, please let us know.

Thanks and regards!

Comment 14 Josh Carter 2018-10-02 14:33:34 UTC
Dear customer, 

The CloudForms team is reviewing the current CloudForms Bug(defect) backlog in order to target engineering efforts. We are closing any bugs for versions that no longer have an active errata stream or that have hit their age limit. We are committing to better management of the backlog as we move forward. If you have an bug that you are still able to reproduce on a current version of CloudForms please open a new bug. 

If you have any concerns about this, please let us know.

Thanks and regards!

Comment 15 Josh Carter 2018-10-02 14:34:59 UTC
Dear customer, 

The CloudForms team is reviewing the current CloudForms Bug(defect) backlog in order to target engineering efforts. We are closing any bugs for versions that no longer have an active errata stream or that have hit their age limit. We are committing to better management of the backlog as we move forward. If you have an bug that you are still able to reproduce on a current version of CloudForms please open a new bug. 

If you have any concerns about this, please let us know.

Thanks and regards!


Note You need to log in before you can comment on or make changes to this bug.