Description of problem:
Customer has two UI appliances running within AWS, behind an AWS load balancer. These can consistently be accessed directly, with no issue. Between client and myself, the LB connection occasionally denies authentication (or other web access) past the login screen.
The logs show CSRF token authentication validation errors.
Allegedly, the LB is configured with sticky sessions, and regardless, both the appliances have the advanced setting "session_store: sql"
Where are you experiencing the behavior? What environment?
Brand new AWS deployment
Version-Release number of selected component (if applicable):
Created attachment 1242295 [details]
Matous, can you try to reproduce this please. Maybe with Azure or GCE LBs as well.
1. Do we have any idea of how their network in AWS is laid out?
2. Are they using multiple private subnets in a VPC?
3. Do they have any specific Network ACL's or Security groups?
4. What are they using internally for DNS and does the ELB point at hostnames or IP's?
I think we need more info to duplicate the environment.
2. There are multiple subnets for VPC, mostly used for high availability.
3. Yes, both of them. ACLs for subnets, Security groups for the rest.
4. When adding ELB they create endpoint with A record. We need to add CNAME record pointing to that record on our domain then.
When I try to create HTTPS ELB they require primary key and cert file.
I tried to evade it by adding TCP rule on port 443, but it ended up with SSL_ERROR_RX_RECORD_TOO_LONG because appliances use IP address and then I tried to access it from endpoint.
Also when not using http/https ELB I can't apply sticky sessions.
I can try to reproduce it with HTTP only, but it probably can have some impact?
I setup an experiment with a Fedora VM doing a HTTPS proxy for an appliance.
I have a PR for that:
I am looking for other problems that would prevent or complicate the use of CFME behing a proxy or load balancer.
I can see that the 'content-security-policy' containts a hard-coded IP of the apliance:
default-src 'self'; connect-src 'self' wss://192.168.122.187; frame-src 'self'; script-src 'unsafe-eval' 'unsafe-inline' 'self'; style-src 'unsafe-inline' 'self'; report-uri /dashboard/csp_report
that will cause issues with HTML5 base VNC/SPICE consoles and notification drawer, but it can be probably handled using header-editing capabilities of Apache proxy.
My proxy config follows:
BalancerMember https://188.8.131.52:443 ping=60 timeout=300 disablereuse=On route=cformsapp0p
# BalancerMember http://192.168.122.1:3000 ping=60 timeout=300 disablereuse=On route=cformsapp0p
RequestHeader set X_FORWARDED_PROTO 'https'
RequestHeader set X-Forwarded-Ssl on
ProxyPass / balancer://cloudforms/ nocanon
ProxyPassReverse / balancer://cloudforms/
Anyway we should carry on trying to reproduce the issue on Amazon.
Matous: regarding "When I try to create HTTPS ELB they require primary key and cert file.": you can generate a primary key and certificate using openssl command line.
The 2 problems we have identified are both fixed: absolute URLs and CSP headers.
I am currently testing the proxying with Apache and HAproxy.
I am putting some notes here: https://github.com/martinpovolny/manageiq-utils/wiki/HA-proxying-ManageIQ-UI-Classic (until I am done and will write an article about this).
I can use some help testing on Amazon.
The CloudForms team is reviewing the current CloudForms Bug(defect) backlog in order to target engineering efforts. We are closing any bugs for versions that no longer have an active errata stream or that have hit their age limit. We are committing to better management of the backlog as we move forward. If you have an bug that you are still able to reproduce on a current version of CloudForms please open a new bug.
If you have any concerns about this, please let us know.
Thanks and regards!