Bug 1411810

Summary: ipa-replica-install fails with 406 Client Error: Key name ca/caSigningCert%20cert-pki-ca does not match subject ca/caSigningCert cert-pki-ca
Product: [Fedora] Fedora Reporter: Jan Pazdziora <jpazdziora>
Component: freeipaAssignee: Christian Heimes <cheimes>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 26CC: abokovoy, cheimes, ipa-maint, jcholast, jhrozek, mbabinsk, pvoborni, rcritten, ssorce
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: freeipa-4.4.4-2.fc26 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1436763 (view as bug list) Environment:
Last Closed: 2017-06-09 19:08:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1436763    

Description Jan Pazdziora 2017-01-10 14:46:19 UTC
Description of problem:

When running ipa-replica-install in a container, the process ends with

Done configuring the web interface (httpd).
Applying LDAP updates
Upgrading IPA:
  [1/9]: stopping directory server
  [2/9]: saving configuration
  [3/9]: disabling listeners
  [4/9]: enabling DS global lock
  [5/9]: starting directory server
  [6/9]: upgrading server
  [7/9]: stopping directory server
  [8/9]: restoring configuration
  [9/9]: starting directory server
Done.
Configuring ipa-otpd
  [1/2]: starting ipa-otpd 
  [2/2]: configuring ipa-otpd to start on boot
Done configuring ipa-otpd.
Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.

ipa.ipapython.install.cli.install_tool(Replica): ERROR    406 Client Error: Key name ca/caSigningCert%20cert-pki-ca does not match subject ca/caSigningCert cert-pki-ca for url: https://ipa.example.test/ipa/keys/ca/caSigningCert%20cert-pki-ca?type=kem&value=eyJhbGciOiJSU0EtT0FFUCIsImVuYyI6IkEyNTZDQkMtSFM1MTIiLCJraWQiOm51bGx9.eu3DOhYXXz8MLWBskVZNSIMKriM80lKAxr47NIU0FMgKqiQdpnIOWL9zFa02-7g1q3obkI79AE60VY3Wiaf1e8jBhg6VZpDSzcC3WYToEWjI4PtipgrjO-NaJMElb6yZOjl0MfWsWcGZ_XtuIMIvjIYTxAV79gebyJhEkof4gj-I2JH5r9Eg4hI3y5UW0C51-6EbUitZLlCu5nGNrzu7eIJ689vaTs-rUrOqbWfhFDY1CRky3JkPrK6O6gjz_ZZch0vyplBOSoZQKNjQ6v6ZXlsfWq96psFvYRnRBqBsWaJZfZ6XUZpClOzzfyVHaGVle1i-BCXa_NnUk_ejfR4X5A.uHCvi1pP2NOQpWqQ3Hdl0w.moz77uszWhRWVjBY1FtsoIJie6P9LpGskkITtZQF5zmw5AuwsX6D_gLoFr00YxbgNZYd9h_dR9lHeabwOaJPHn-a3ZIEWEbukDpgqdPH-_YFiGFbAUOHfLS1omiMdy6HfFNpkId5v3A0NSSRXhlzZLNN5654oNOiFEW6DC4im8zom12TS0E6lyfjLyb8eFFAg7UqUqmBH4OtEBJo6777QPm6kBgfAwO6rMOV06uHzaP6yyDy5D14c-Zd9Y_-knWmzxGo-0B8MHdWHn0_5vYyYPazSy1H3x-nlCAzRDAFrEHlbEoXSf9Lx9J2lvzHSmst6DbXaWqDO5mFjFRlXElwZvxegpYCBhDO_kwPWNEUocwqAZyGzELtxuaCVu5RQq514ueFKNNgwyknIn5aZ_MXJXk3D0PlGi5eCzkhccV2RYltQZ1chRUUZzIjq4doCm9uIHj1aRoxLBz43RGLGSJrjvl8as9y6W_T-6SsPuREdZpQZigTXPajxo5V0_UwVxIpKYAjjJS6MMTXUCZSXFhCKO21hYTh1iopkVCIKEE4yHl8g3k18v9XRyi14Pbf_cBAEeoPjOI_W-RY5sTglKPftphtVNTT5dy7wP6oLdpa22dfG8qgevKGCrxpG1Gnqw1865ULy1SXGNYHMMGOd7o5_gQ3FE5WfRkKqnkQ2YNFYWo.MU1o2NgIww9haxlDRJsjAN5opFrI6i2hu1qAH4RfiPM
ipa.ipapython.install.cli.install_tool(Replica): ERROR    The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information

Version-Release number of selected component (if applicable):

freeipa-server-4.4.3-2.fc26.x86_64

How reproducible:

Seen once, assume deterministic.

Steps to Reproduce:
1. Have existing IPA master, for example in a container.
2. Run docker run -t --name freeipa-replica-container -h replica.example.test --tmpfs /run --tmpfs /tmp -v /dev/urandom:/dev/random:ro -v /opt/ipa-replica-fedora-rawhide:/data -v /sys/fs/cgroup:/sys/fs/cgroup:ro  --link freeipa-server-container:ipa.example.test --net freeipa-network --net-alias replica.example.test --cap-add=SYS_TIME -e IPA_SERVER_INSTALL_OPTS='' freeipa-server
with ipa-replica-install-options containing
-U
--skip-conncheck
--principal admin
--password Secret123
--setup-ca
--server ipa.example.test
--domain example.test

Actual results:

Configuring client side components
Client hostname: replica.example.test
Realm: EXAMPLE.TEST
DNS Domain: example.test
IPA Server: ipa.example.test
BaseDN: dc=example,dc=test

Skipping synchronizing time with NTP server.
Successfully retrieved CA cert
    Subject:     CN=Certificate Authority,O=EXAMPLE.TEST
    Issuer:      CN=Certificate Authority,O=EXAMPLE.TEST
    Valid From:  Tue Jan 10 14:15:40 2017 UTC
    Valid Until: Sat Jan 10 14:15:40 2037 UTC

Enrolled in IPA realm EXAMPLE.TEST
Created /etc/ipa/default.conf
New SSSD config will be created
Configured sudoers in /etc/nsswitch.conf
Configured /etc/sssd/sssd.conf
Configured /etc/krb5.conf for IPA realm EXAMPLE.TEST
trying https://ipa.example.test/ipa/json
Forwarding 'schema' to json server 'https://ipa.example.test/ipa/json'
trying https://ipa.example.test/ipa/json
Forwarding 'ping' to json server 'https://ipa.example.test/ipa/json'
Forwarding 'ca_is_enabled' to json server 'https://ipa.example.test/ipa/json'
Systemwide CA database updated.
SSSD enabled
Configured /etc/openldap/ldap.conf
/etc/ssh/ssh_config not found, skipping configuration
/etc/ssh/sshd_config not found, skipping configuration
Configuring example.test as NIS domain.
Client configuration complete.

ipa         : ERROR    The host name ipa.example.test does not match the value freeipa-server-container.freeipa-network obtained by reverse lookup on IP address 172.18.0.2
Configuring NTP daemon (ntpd)
  [1/4]: stopping ntpd
  [2/4]: writing configuration
  [3/4]: configuring ntpd to start on boot
  [4/4]: starting ntpd
Done configuring NTP daemon (ntpd).
Configuring directory server (dirsrv). Estimated time: 1 minute
  [1/44]: creating directory server user
  [2/44]: creating directory server instance
  [3/44]: updating configuration in dse.ldif
  [4/44]: restarting directory server
  [5/44]: adding default schema
  [6/44]: enabling memberof plugin
  [7/44]: enabling winsync plugin
  [8/44]: configuring replication version plugin
  [9/44]: enabling IPA enrollment plugin
  [10/44]: enabling ldapi
  [11/44]: configuring uniqueness plugin
  [12/44]: configuring uuid plugin
  [13/44]: configuring modrdn plugin
  [14/44]: configuring DNS plugin
  [15/44]: enabling entryUSN plugin
  [16/44]: configuring lockout plugin
  [17/44]: configuring topology plugin
  [18/44]: creating indices
  [19/44]: enabling referential integrity plugin
  [20/44]: configuring certmap.conf
  [21/44]: configure autobind for root
  [22/44]: configure new location for managed entries
  [23/44]: configure dirsrv ccache
  [24/44]: enabling SASL mapping fallback
  [25/44]: restarting directory server
  [26/44]: creating DS keytab
  [27/44]: retrieving DS Certificate
  [28/44]: restarting directory server
  [29/44]: setting up initial replication
Starting replication, please wait until this has completed.

Update in progress, 1 seconds elapsed
Update in progress, 2 seconds elapsed
Update in progress, 3 seconds elapsed
Update succeeded

  [30/44]: adding sasl mappings to the directory
  [31/44]: updating schema
  [32/44]: setting Auto Member configuration
  [33/44]: enabling S4U2Proxy delegation
  [34/44]: importing CA certificates from LDAP
  [35/44]: initializing group membership
  [36/44]: adding master entry
  [37/44]: initializing domain level
  [38/44]: configuring Posix uid/gid generation
  [39/44]: adding replication acis
  [40/44]: enabling compatibility plugin
  [41/44]: activating sidgen plugin
  [42/44]: activating extdom plugin
  [43/44]: tuning directory server
  [44/44]: configuring directory to start on boot
Done configuring directory server (dirsrv).
Configuring ipa-custodia
  [1/5]: Generating ipa-custodia config file
  [2/5]: Generating ipa-custodia keys
  [3/5]: Importing RA Key
  [4/5]: starting ipa-custodia 
  [5/5]: configuring ipa-custodia to start on boot
Done configuring ipa-custodia.

MARK-LWD-LOOP -- 2017-01-10 09:22:30 --
Configuring Kerberos KDC (krb5kdc). Estimated time: 30 seconds
  [1/4]: configuring KDC
  [2/4]: adding the password extension to the directory
  [3/4]: starting the KDC
  [4/4]: configuring KDC to start on boot
Done configuring Kerberos KDC (krb5kdc).
Configuring kadmin
  [1/2]: starting kadmin 
  [2/2]: configuring kadmin to start on boot
Done configuring kadmin.
Configuring ipa_memcached
  [1/2]: starting ipa_memcached 
  [2/2]: configuring ipa_memcached to start on boot
Done configuring ipa_memcached.
Configuring the web interface (httpd). Estimated time: 1 minute
  [1/20]: setting mod_nss port to 443
  [2/20]: setting mod_nss cipher suite
  [3/20]: setting mod_nss protocol list to TLSv1.0 - TLSv1.2
  [4/20]: setting mod_nss password file
  [5/20]: enabling mod_nss renegotiate
  [6/20]: adding URL rewriting rules
  [7/20]: configuring httpd
  [8/20]: configure certmonger for renewals
  [9/20]: setting up httpd keytab
  [10/20]: setting up ssl
  [11/20]: importing CA certificates from LDAP
  [12/20]: publish CA cert
  [13/20]: clean up any existing httpd ccache
  [14/20]: configuring SELinux for httpd
  [15/20]: create KDC proxy user
  [16/20]: create KDC proxy config
  [17/20]: enable KDC proxy
  [18/20]: restarting httpd
  [19/20]: configuring httpd to start on boot
  [20/20]: enabling oddjobd
Done configuring the web interface (httpd).
Applying LDAP updates
Upgrading IPA:
  [1/9]: stopping directory server
  [2/9]: saving configuration
  [3/9]: disabling listeners
  [4/9]: enabling DS global lock
  [5/9]: starting directory server
  [6/9]: upgrading server
  [7/9]: stopping directory server
  [8/9]: restoring configuration
  [9/9]: starting directory server
Done.
Configuring ipa-otpd
  [1/2]: starting ipa-otpd 
  [2/2]: configuring ipa-otpd to start on boot
Done configuring ipa-otpd.
Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.

ipa.ipapython.install.cli.install_tool(Replica): ERROR    406 Client Error: Key name ca/caSigningCert%20cert-pki-ca does not match subject ca/caSigningCert cert-pki-ca for url: https://ipa.example.test/ipa/keys/ca/caSigningCert%20cert-pki-ca?type=kem&value=eyJhbGciOiJSU0EtT0FFUCIsImVuYyI6IkEyNTZDQkMtSFM1MTIiLCJraWQiOm51bGx9.eu3DOhYXXz8MLWBskVZNSIMKriM80lKAxr47NIU0FMgKqiQdpnIOWL9zFa02-7g1q3obkI79AE60VY3Wiaf1e8jBhg6VZpDSzcC3WYToEWjI4PtipgrjO-NaJMElb6yZOjl0MfWsWcGZ_XtuIMIvjIYTxAV79gebyJhEkof4gj-I2JH5r9Eg4hI3y5UW0C51-6EbUitZLlCu5nGNrzu7eIJ689vaTs-rUrOqbWfhFDY1CRky3JkPrK6O6gjz_ZZch0vyplBOSoZQKNjQ6v6ZXlsfWq96psFvYRnRBqBsWaJZfZ6XUZpClOzzfyVHaGVle1i-BCXa_NnUk_ejfR4X5A.uHCvi1pP2NOQpWqQ3Hdl0w.moz77uszWhRWVjBY1FtsoIJie6P9LpGskkITtZQF5zmw5AuwsX6D_gLoFr00YxbgNZYd9h_dR9lHeabwOaJPHn-a3ZIEWEbukDpgqdPH-_YFiGFbAUOHfLS1omiMdy6HfFNpkId5v3A0NSSRXhlzZLNN5654oNOiFEW6DC4im8zom12TS0E6lyfjLyb8eFFAg7UqUqmBH4OtEBJo6777QPm6kBgfAwO6rMOV06uHzaP6yyDy5D14c-Zd9Y_-knWmzxGo-0B8MHdWHn0_5vYyYPazSy1H3x-nlCAzRDAFrEHlbEoXSf9Lx9J2lvzHSmst6DbXaWqDO5mFjFRlXElwZvxegpYCBhDO_kwPWNEUocwqAZyGzELtxuaCVu5RQq514ueFKNNgwyknIn5aZ_MXJXk3D0PlGi5eCzkhccV2RYltQZ1chRUUZzIjq4doCm9uIHj1aRoxLBz43RGLGSJrjvl8as9y6W_T-6SsPuREdZpQZigTXPajxo5V0_UwVxIpKYAjjJS6MMTXUCZSXFhCKO21hYTh1iopkVCIKEE4yHl8g3k18v9XRyi14Pbf_cBAEeoPjOI_W-RY5sTglKPftphtVNTT5dy7wP6oLdpa22dfG8qgevKGCrxpG1Gnqw1865ULy1SXGNYHMMGOd7o5_gQ3FE5WfRkKqnkQ2YNFYWo.MU1o2NgIww9haxlDRJsjAN5opFrI6i2hu1qAH4RfiPM
ipa.ipapython.install.cli.install_tool(Replica): ERROR    The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information
FreeIPA server configuration failed.

The /var/log/ipareplica-install.log ends with

2017-01-10T14:23:59Z DEBUG Starting external process
2017-01-10T14:23:59Z DEBUG args=/bin/systemctl is-active ipa-otpd.socket
2017-01-10T14:23:59Z DEBUG Process finished, return code=0
2017-01-10T14:23:59Z DEBUG stdout=active

2017-01-10T14:23:59Z DEBUG stderr=
2017-01-10T14:23:59Z DEBUG   duration: 0 seconds
2017-01-10T14:23:59Z DEBUG   [2/2]: configuring ipa-otpd to start on boot
2017-01-10T14:23:59Z DEBUG Starting external process
2017-01-10T14:23:59Z DEBUG args=/bin/systemctl is-enabled ipa-otpd.socket
2017-01-10T14:23:59Z DEBUG Process finished, return code=1
2017-01-10T14:23:59Z DEBUG stdout=disabled

2017-01-10T14:23:59Z DEBUG stderr=
2017-01-10T14:23:59Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2017-01-10T14:23:59Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state'
2017-01-10T14:23:59Z DEBUG Starting external process
2017-01-10T14:23:59Z DEBUG args=/bin/systemctl disable ipa-otpd.socket
2017-01-10T14:23:59Z DEBUG Process finished, return code=0
2017-01-10T14:23:59Z DEBUG stdout=
2017-01-10T14:23:59Z DEBUG stderr=
2017-01-10T14:23:59Z DEBUG flushing ldap://replica.example.test:389 from SchemaCache
2017-01-10T14:23:59Z DEBUG retrieving schema for SchemaCache url=ldap://replica.example.test:389 conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f784249e998>
2017-01-10T14:23:59Z DEBUG   duration: 0 seconds
2017-01-10T14:23:59Z DEBUG Done configuring ipa-otpd.
2017-01-10T14:23:59Z DEBUG   File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 172, in execute
    return_value = self.run()
  File "/usr/lib/python2.7/site-packages/ipapython/install/cli.py", line 318, in run
    cfgr.run()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 310, in run
    self.execute()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 334, in execute
    for nothing in self._executor():
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 376, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 405, in _handle_execute_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 395, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 366, in __runner
    step()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 363, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 597, in _configure
    next(executor)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 376, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 405, in _handle_execute_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 460, in _handle_exception
    self.__parent._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 395, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 457, in _handle_exception
    super(ComponentBase, self)._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 395, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 366, in __runner
    step()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 363, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python2.7/site-packages/ipapython/install/common.py", line 63, in _install
    for nothing in self._installer(self.parent):
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/replicainstall.py", line 1727, in main
    promote(self)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/replicainstall.py", line 367, in decorated
    func(installer)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/replicainstall.py", line 1516, in promote
    custodia.get_ca_keys(config.ca_host_name, ca_data[0], ca_data[1])
  File "/usr/lib/python2.7/site-packages/ipaserver/install/custodiainstance.py", line 182, in get_ca_keys
    self.__get_keys(ca_host, cacerts_file, cacerts_pwd, data)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/custodiainstance.py", line 143, in __get_keys
    value = cli.fetch_key(os.path.join(prefix, nickname), False)
  File "/usr/lib/python2.7/site-packages/ipapython/secrets/client.py", line 98, in fetch_key
    r.raise_for_status()
  File "/usr/lib/python2.7/site-packages/requests/models.py", line 893, in raise_for_status
    raise HTTPError(http_error_msg, response=self)

2017-01-10T14:23:59Z DEBUG The ipa-replica-install command failed, exception: HTTPError: 406 Client Error: Key name ca/caSigningCert%20cert-pki-ca does not match subject ca/caSigningCert cert-pki-ca for url: https://ipa.example.test/ipa/keys/ca/caSigningCert%20cert-pki-ca?type=kem&value=eyJhbGciOiJSU0EtT0FFUCIsImVuYyI6IkEyNTZDQkMtSFM1MTIiLCJraWQiOm51bGx9.eu3DOhYXXz8MLWBskVZNSIMKriM80lKAxr47NIU0FMgKqiQdpnIOWL9zFa02-7g1q3obkI79AE60VY3Wiaf1e8jBhg6VZpDSzcC3WYToEWjI4PtipgrjO-NaJMElb6yZOjl0MfWsWcGZ_XtuIMIvjIYTxAV79gebyJhEkof4gj-I2JH5r9Eg4hI3y5UW0C51-6EbUitZLlCu5nGNrzu7eIJ689vaTs-rUrOqbWfhFDY1CRky3JkPrK6O6gjz_ZZch0vyplBOSoZQKNjQ6v6ZXlsfWq96psFvYRnRBqBsWaJZfZ6XUZpClOzzfyVHaGVle1i-BCXa_NnUk_ejfR4X5A.uHCvi1pP2NOQpWqQ3Hdl0w.moz77uszWhRWVjBY1FtsoIJie6P9LpGskkITtZQF5zmw5AuwsX6D_gLoFr00YxbgNZYd9h_dR9lHeabwOaJPHn-a3ZIEWEbukDpgqdPH-_YFiGFbAUOHfLS1omiMdy6HfFNpkId5v3A0NSSRXhlzZLNN5654oNOiFEW6DC4im8zom12TS0E6lyfjLyb8eFFAg7UqUqmBH4OtEBJo6777QPm6kBgfAwO6rMOV06uHzaP6yyDy5D14c-Zd9Y_-knWmzxGo-0B8MHdWHn0_5vYyYPazSy1H3x-nlCAzRDAFrEHlbEoXSf9Lx9J2lvzHSmst6DbXaWqDO5mFjFRlXElwZvxegpYCBhDO_kwPWNEUocwqAZyGzELtxuaCVu5RQq514ueFKNNgwyknIn5aZ_MXJXk3D0PlGi5eCzkhccV2RYltQZ1chRUUZzIjq4doCm9uIHj1aRoxLBz43RGLGSJrjvl8as9y6W_T-6SsPuREdZpQZigTXPajxo5V0_UwVxIpKYAjjJS6MMTXUCZSXFhCKO21hYTh1iopkVCIKEE4yHl8g3k18v9XRyi14Pbf_cBAEeoPjOI_W-RY5sTglKPftphtVNTT5dy7wP6oLdpa22dfG8qgevKGCrxpG1Gnqw1865ULy1SXGNYHMMGOd7o5_gQ3FE5WfRkKqnkQ2YNFYWo.MU1o2NgIww9haxlDRJsjAN5opFrI6i2hu1qAH4RfiPM
2017-01-10T14:23:59Z ERROR 406 Client Error: Key name ca/caSigningCert%20cert-pki-ca does not match subject ca/caSigningCert cert-pki-ca for url: https://ipa.example.test/ipa/keys/ca/caSigningCert%20cert-pki-ca?type=kem&value=eyJhbGciOiJSU0EtT0FFUCIsImVuYyI6IkEyNTZDQkMtSFM1MTIiLCJraWQiOm51bGx9.eu3DOhYXXz8MLWBskVZNSIMKriM80lKAxr47NIU0FMgKqiQdpnIOWL9zFa02-7g1q3obkI79AE60VY3Wiaf1e8jBhg6VZpDSzcC3WYToEWjI4PtipgrjO-NaJMElb6yZOjl0MfWsWcGZ_XtuIMIvjIYTxAV79gebyJhEkof4gj-I2JH5r9Eg4hI3y5UW0C51-6EbUitZLlCu5nGNrzu7eIJ689vaTs-rUrOqbWfhFDY1CRky3JkPrK6O6gjz_ZZch0vyplBOSoZQKNjQ6v6ZXlsfWq96psFvYRnRBqBsWaJZfZ6XUZpClOzzfyVHaGVle1i-BCXa_NnUk_ejfR4X5A.uHCvi1pP2NOQpWqQ3Hdl0w.moz77uszWhRWVjBY1FtsoIJie6P9LpGskkITtZQF5zmw5AuwsX6D_gLoFr00YxbgNZYd9h_dR9lHeabwOaJPHn-a3ZIEWEbukDpgqdPH-_YFiGFbAUOHfLS1omiMdy6HfFNpkId5v3A0NSSRXhlzZLNN5654oNOiFEW6DC4im8zom12TS0E6lyfjLyb8eFFAg7UqUqmBH4OtEBJo6777QPm6kBgfAwO6rMOV06uHzaP6yyDy5D14c-Zd9Y_-knWmzxGo-0B8MHdWHn0_5vYyYPazSy1H3x-nlCAzRDAFrEHlbEoXSf9Lx9J2lvzHSmst6DbXaWqDO5mFjFRlXElwZvxegpYCBhDO_kwPWNEUocwqAZyGzELtxuaCVu5RQq514ueFKNNgwyknIn5aZ_MXJXk3D0PlGi5eCzkhccV2RYltQZ1chRUUZzIjq4doCm9uIHj1aRoxLBz43RGLGSJrjvl8as9y6W_T-6SsPuREdZpQZigTXPajxo5V0_UwVxIpKYAjjJS6MMTXUCZSXFhCKO21hYTh1iopkVCIKEE4yHl8g3k18v9XRyi14Pbf_cBAEeoPjOI_W-RY5sTglKPftphtVNTT5dy7wP6oLdpa22dfG8qgevKGCrxpG1Gnqw1865ULy1SXGNYHMMGOd7o5_gQ3FE5WfRkKqnkQ2YNFYWo.MU1o2NgIww9haxlDRJsjAN5opFrI6i2hu1qAH4RfiPM
2017-01-10T14:23:59Z ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information

Expected results:

No error, replica set up.

Additional info:

I currently do not have capacity to reproduce outside of containers.

Note that the setup is without DNS servers, due to bug 1403352.

Comment 2 Martin Babinsky 2017-01-19 14:48:29 UTC
It seems that the root cause is in custodia client escaping whitespace in the request URI when fetching CA keys from remote master.

I have tried to backport custodia-0.2.0-2.fc26.noarch to F25 to see if the rebase causes this issue but replica install passed fine. I suspect that maybe one of the dependencies in rawhide (python-requests-2.12.4-3.fc26 or python-urllib3-1.19.1-2) may be to blame. Christian, can you look into this issue?

Relevant data ipareplica-install.log:

{{{
 File "/usr/lib/python2.7/site-packages/ipapython/install/common.py", line 63, in _install
    for nothing in self._installer(self.parent):
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/replicainstall.py", line 1727, in main
    promote(self)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/replicainstall.py", line 367, in decorated
    func(installer)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/replicainstall.py", line 1516, in promote
    custodia.get_ca_keys(config.ca_host_name, ca_data[0], ca_data[1])
  File "/usr/lib/python2.7/site-packages/ipaserver/install/custodiainstance.py", line 182, in get_ca_keys
    self.__get_keys(ca_host, cacerts_file, cacerts_pwd, data)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/custodiainstance.py", line 143, in __get_keys
    value = cli.fetch_key(os.path.join(prefix, nickname), False)
  File "/usr/lib/python2.7/site-packages/ipapython/secrets/client.py", line 98, in fetch_key
    r.raise_for_status()
  File "/usr/lib/python2.7/site-packages/requests/models.py", line 893, in raise_for_status
    raise HTTPError(http_error_msg, response=self)

2017-01-19T10:37:48Z DEBUG The ipa-replica-install command failed, exception: HTTPError: 406 Client Error: Key name ca/caSigningCert%20cert-pki-ca does not match subject ca/caSigningCert cert-pki-ca for url: https://ipa.example.test/ipa/keys/ca/caSigningCert%20cert-pki-ca?type=kem&value=eyJhbGciOiJSU0EtT0FFUCIsImVuYyI6IkEyNTZDQkMtSFM1MTIiLCJraWQiOm51bGx9.RJ7WeWvjUL0TP0GJQXt5jiEoOekZcUYuYb9XjinxPo81W4-0n1XUvIFShyrxqHvGAvziW6pPC6QKeAf7UnZ0-h8OBR48jGhNq1RK9lch4VBwA47zPU2dfgAVhlhhZgxV0EyUHYaZ79JUlX6GpADsTooWk0qG6sPKAVm9qLqKZm9t3qMRZjItXR2G1wzVsYHej4nyPj-_kBXfX9N4fZGApQa_9YfN-l2Ulhzow_xUnzuyh8tSM0ead9MVk5S9Xo2fhI2finvDOQN7od1md_u7q3us9UJngWJtplwJ4ePLoTt5T5pgEBiIfPLMwbYOpalrzCHnLSHlqosKrMjVUl5zOg.NyiW-h8Q5xf0FaPGi9AGcQ.Gqlh3vPB24U9eeR9JfqFFsYYNs4HZtPvPJnDyD0bcGFJV2KwHXwppa2AUs-R7WJKlmCGQStYGvrADyGTeU2eb4-o34vCxE6yaRI4orwqy3sHsTHLzyvzg62bQAOC-40FkebGAcXsKIS1IdixIWsxkl89Ie0OvqMQdfY7ah3MOOMXfl3grzVWWccExCdKabggSD4tDFgVPmrZbkxEFWGuNfc4yocs3bgA6FOcal5u7NaglSsZGgDlSxe-L9fdk1ifc6pgQBvgKRku-DoWxQLuFxqfO5nvSk_HyDudi5EYtsnCiFFWji0uPJokS21E425fSKm9nJTLi3vR1Ufe9Dn1EI0EjiA1I5d_MvivoR7Hp0CofHNryzRua6gdv2PU7ERx3udKNcb5g-pMTkT_LPNtl0zHs9LD3nconrbikbjSecFiTtp2MZn2OVdtCaW2Sy84A-fAbRk3TXV-Ay1XlsWbalTWLIgSVpunKP37ySJDEqa40hLIyy7XK3Y2jhGYZcRGhnV0dLPRvXUrERxNizyEd5UfrxLz_3p3Ki4xeWdpzOEl3hvgEHQnJCwmtyEwsaAe4TJCzL78gZhGmH5_jSrQHbJV7N1-HgqT8PHbuzXVvospM34eP56rPZD9lRlEH9HxvurI32ZuBBOQRChdZ5kIrBIwBlQyHjsoKKZ8k1xiZBo.0w3wD3GELRyioUVb-0fJB1N3R9OLS1xpF6oC_xIcV98
}}}

ipa-custodia.audit.log on master:

{{{
2017-01-19 10:36:11 - SimpleCredsAuth-[auth:simple]    - PASS: '656' authenticated as '48, 48'
2017-01-19 10:36:11 - SimpleHeaderAuth-[auth:header]   - PASS: '656' authenticated as '(null)'
2017-01-19 10:36:11 - IPAKEMKeys-[authz:kemkeys]       - PASS: '656' authorized for '/keys'
2017-01-19 10:36:11 - Secrets-[/keys]                  - ALLOWED: '(null)' requested key 'ra/ipaCert'
2017-01-19 10:37:48 - SimpleCredsAuth-[auth:simple]    - PASS: '652' authenticated as '48, 48'
2017-01-19 10:37:48 - SimpleHeaderAuth-[auth:header]   - PASS: '652' authenticated as '(null)'
2017-01-19 10:37:48 - IPAKEMKeys-[authz:kemkeys]       - PASS: '652' authorized for '/keys'
2017-01-19 10:37:48 - Secrets-[/keys]                  - DENIED: '(null)' requested key 'ca/caSigningCert%20cert-pki-ca
}}}

Comment 3 Christian Heimes 2017-01-20 09:46:26 UTC
We are aware that non-numeric characters like spaces and non-ASCII chars are not handled well by Custodia. I recently removed some unquoting from Custodia, e.g. https://github.com/latchset/custodia/commit/9dd4ca48cae2f09abed3226d1b20a00ff843fb89

I'll try to find some time to investigate the issue. I'm planning to release a new version of Custodia soonish anyway.

Comment 4 Petr Vobornik 2017-02-17 17:21:16 UTC
Upstream ticket:
https://fedorahosted.org/freeipa/ticket/6688

Comment 5 Fedora End Of Life 2017-02-28 10:55:19 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 26 development cycle.
Changing version to '26'.

Comment 6 Christian Heimes 2017-03-28 14:08:01 UTC
Upstream bug: https://github.com/latchset/custodia/issues/135

The bug has been fixed by PR https://github.com/latchset/custodia/pull/139 and has landed in Custodia release 0.3.1. I'm working on releases for F26 and rawhide.

Comment 7 Petr Vobornik 2017-03-28 15:50:57 UTC
Dependencies bumped in:
ipa-4-5:

    403263df7a3be61086c87c5577698cf32a912065 Use Custodia 0.3.1 features

master:

    f5bf5466eda0de2a211b4f2682e5c50b82577701 Use Custodia 0.3.1 features

Comment 8 Fedora Update System 2017-05-23 11:14:50 UTC
freeipa-4.4.4-2.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-0d0ec3e0af

Comment 9 Fedora Update System 2017-05-23 18:16:31 UTC
freeipa-4.4.4-2.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-0d0ec3e0af

Comment 10 Fedora Update System 2017-06-01 18:05:26 UTC
custodia-0.3.1-2.fc26 freeipa-4.4.4-2.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-0d0ec3e0af

Comment 11 Fedora Update System 2017-06-04 19:38:31 UTC
custodia-0.3.1-2.fc26, freeipa-4.4.4-2.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-0d0ec3e0af

Comment 12 Fedora Update System 2017-06-09 19:08:27 UTC
custodia-0.3.1-2.fc26, freeipa-4.4.4-2.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.