Fixed in mod_auth_mellon-0.13.1-2.el7_5 by the addition of the MellonSignatureMethod config directive.

Verified.

Version::
mod_auth_mellon-0.13.1-2.el7_5.x86_64

Results:

* Setup ADFS on my ADDC which is also my ADCS.

# generate cert for Service Provider



[root@sp1 adfs]# openssl req -new -newkey rsa:2048 -keyout mellon.key -nodes -out mellon.csr -subj '/CN=sp1.keycloak.test'
Generating a 2048 bit RSA private key
.+++
...................................................................+++
writing new private key to 'mellon.key'
-----

[root@sp1 adfs]# scp mellon.csr Administrator.test:/cygdrive/c/certs
Administrator.test's password: 
mellon.csr                                                  100%  903   219.8KB/s   00:00    


# Sign the request with AD CS and copy back to SP

[root@sp1 adfs]# scp Administrator.test:/cygdrive/c/certs/mellon.crt .
Administrator.test's password: 
mellon.crt                                                  100% 1786   329.4KB/s   00:00    

[root@sp1 adfs]# cp mellon.crt /etc/pki/tls/certs
[root@sp1 adfs]# cp mellon.key /etc/pki/tls/private
[root@sp1 adfs]# vim /etc/httpd/conf.d/ssl.conf 
# Set following:
ServerName sp1.keycloak.test
...
SSLCertificateFile /etc/pki/tls/certs/mellon.crt
...
SSLCertificateKeyFile /etc/pki/tls/private/mellon.key


# Prep Apache for SAML

[root@sp1 ~]# mkdir /var/www/html/{mellon,private}
[root@sp1 ~]# mkdir /etc/httpd/saml2
[root@sp1 ~]# cd /etc/httpd/saml2

# create metadata file and cert/key

[root@sp1 saml2]# /usr/libexec/mod_auth_mellon/mellon_create_metadata.sh "https://sp1.keycloak.test" "https://sp1.keycloak.test/mellon"
Output files:
Private key:                              https_sp_.keycloak.test.key
Certificate:                              https_sp_.keycloak.test.cert
Metadata:                                 https_sp_.keycloak.test.xml
Host:                                     sp1.keycloak.test

Endpoints:
SingleLogoutService (SOAP):               https://sp1.keycloak.test/mellon/logout
SingleLogoutService (HTTP-Redirect):      https://sp1.keycloak.test/mellon/logout
AssertionConsumerService (HTTP-POST):     https://sp1.keycloak.test/mellon/postResponse
AssertionConsumerService (HTTP-Artifact): https://sp1.keycloak.test/mellon/artifactResponse
AssertionConsumerService (PAOS):          https://sp1.keycloak.test/mellon/paosResponse

# Download ADFS metadata

[root@sp1 saml2]# curl -kL -o https_win1.ad.test.xml https://win1.ad.test/FederationMetadata/2007-06/FederationMetadata.xml
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 71029    0 71029    0     0   500k      0 --:--:-- --:--:-- --:--:--  502k

[root@sp1 saml2]# vim /etc/httpd/conf.d/auth_mellon.conf
[root@sp1 ~]# cat /etc/httpd/conf.d/auth_mellon.conf
MellonCacheSize 100
MellonLockFile "/run/mod_auth_mellon/lock"


<Location />
  MellonEnable info
  MellonEndpointPath /mellon/

  MellonSignatureMethod rsa-sha256

  # The mellon metadata
  MellonSPMetadataFile /etc/httpd/saml2/https_sp_.keycloak.test.xml
  MellonSPPrivateKeyFile /etc/httpd/saml2/https_sp_.keycloak.test.key
  MellonSPCertFile /etc/httpd/saml2/https_sp_.keycloak.test.cert

  # The ADFS metadata
  MellonIdPMetadataFile /etc/httpd/saml2/https_win1.ad.test.xml
</Location>

<Location /private>
  AuthType Mellon
  MellonEnable auth
  Require valid-user
</Location>

[root@sp1 conf.d]# systemctl start httpd.service 


# Note MellonSignatureMethod above is set to rsa-sha256.  

# ADFS Relying Party Trust Signature Algorithm set to SHA-256 by default.

# testing by accessing https://sp1.keycloak.test/private/.  If it works, I will see login and be able to sign in with an AD user account in the domain.

# logout with https://sp1.keycloak.test/mellon/logout?ReturnTo=https://sp1.keycloak.test/


So, testing with:

* Mellon=SHA-256 ADFS=SHA-256 ::  PASS
* Mellon=SHA-256 ADFS=SHA-1 :: PASS
* Mellon=SHA-1 ADFS=SHA-1 :: PASS
* Mellon=SHA-1 ADFS=SHA-256 :: FAIL (expected)


Note to change mellon to SHA-1 I changed this line in /etc/httpd/conf.d/auth_mellon.conf:

  MellonSignatureMethod rsa-sha1


As noted above, it is currently expected that the scenario where ADFS is using SHA-256 and mod_auth_mellon is using rsa-sha1 will not work.  The important thing is that Mellon does work when the signing algorithm matches and that it can match SHA256 now.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:1387