Description of problem: Intermittent failure to enroll a system, due to "Internal Server Error" returned from https://f0.cockpit.lan/ipa/json Version-Release number of selected component (if applicable): freeipa-client-4.1.3-3.fc22.x86_64 freeipa-server-4.1.3-2.fc21.x86_64 How reproducible: On my machine 1 in 5 runs of the Cockpit CI system. Actual results: Mär 25 10:12:10 Hostname: x0.cockpit.lan Mär 25 10:12:10 Realm: COCKPIT.LAN Mär 25 10:12:10 DNS Domain: cockpit.lan Mär 25 10:12:10 IPA Server: f0.cockpit.lan Mär 25 10:12:10 BaseDN: dc=cockpit,dc=lan Mär 25 10:12:10 Synchronizing time with KDC... Mär 25 10:12:10 Attempting to sync time using ntpd. Will timeout after 15 seconds Mär 25 10:12:11 Successfully retrieved CA cert Mär 25 10:12:11 Subject: CN=Certificate Authority,O=COCKPIT.LAN Mär 25 10:12:11 Issuer: CN=Certificate Authority,O=COCKPIT.LAN Mär 25 10:12:11 Valid From: Mon Mar 09 11:06:52 2015 UTC Mär 25 10:12:11 Valid Until: Fri Mar 09 11:06:52 2035 UTC Mär 25 10:12:11 Mär 25 10:12:13 Enrolled in IPA realm COCKPIT.LAN Mär 25 10:12:13 Created /etc/ipa/default.conf Mär 25 10:12:14 New SSSD config will be created Mär 25 10:12:14 Configured sudoers in /etc/nsswitch.conf Mär 25 10:12:14 Configured /etc/sssd/sssd.conf Mär 25 10:12:14 Configured /etc/krb5.conf for IPA realm COCKPIT.LAN Mär 25 10:12:14 trying https://f0.cockpit.lan/ipa/json Mär 25 10:12:14 Forwarding 'ping' to json server 'https://f0.cockpit.lan/ipa/json' Mär 25 10:12:14 Cannot connect to the server due to generic error: cannot connect to 'https://f0.cockpit.lan/ipa/json': Internal Server Error Mär 25 10:12:14 Installation failed. Rolling back changes. Mär 25 10:12:15 Failed to list certificates in /etc/ipa/nssdb: Command ''/usr/bin/certutil' '-d' '/etc/ipa/nssdb' '-L'' returned non-zero exit status 255 Mär 25 10:12:15 Failed to remove /etc/ipa/nssdb/cert8.db: [Errno 2] No such file or directory: '/etc/ipa/nssdb/cert8.db' Mär 25 10:12:15 Failed to remove /etc/ipa/nssdb/key3.db: [Errno 2] No such file or directory: '/etc/ipa/nssdb/key3.db' Mär 25 10:12:15 Failed to remove /etc/ipa/nssdb/secmod.db: [Errno 2] No such file or directory: '/etc/ipa/nssdb/secmod.db' Mär 25 10:12:15 Failed to remove /etc/ipa/nssdb/pwdfile.txt: [Errno 2] No such file or directory: '/etc/ipa/nssdb/pwdfile.txt' Mär 25 10:12:15 Unenrolling client from IPA server Mär 25 10:12:16 Removing Kerberos service principals from /etc/krb5.keytab Mär 25 10:12:16 Disabling client Kerberos and LDAP configurations Mär 25 10:12:16 Redundant SSSD configuration file /etc/sssd/sssd.conf was moved to /etc/sssd/sssd.conf.deleted Mär 25 10:12:16 Restoring client configuration files Mär 25 10:12:16 nscd daemon is not installed, skip configuration Mär 25 10:12:16 nslcd daemon is not installed, skip configuration Mär 25 10:12:16 Mär 25 10:12:16 Client uninstall complete. Expected results: Machine joined successfully.
Messages from the IPA server journal: -- Logs begin at Mo 2015-03-09 11:57:56 CET, end at Mi 2015-03-25 10:14:10 CET. -- Mär 25 10:12:13 f0.cockpit.lan ipa-server-guard[803]: Traceback (most recent call last): File "/usr/libexec/certmonger/ipa-server-guard", line 51, in <module> sys.exit(main()) File "/usr/libexec/certmonger/ipa-server-guard", line 41, in main env=os.environ) File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 321, in run preexec_fn=preexec_fn) File "/usr/lib64/python2.7/subprocess.py", line 710, in __init__ errread, errwrite) File "/usr/lib64/python2.7/subprocess.py", line 1223, in _execute_child self.pid = os.fork() OSError: [Errno 12] Cannot allocate memory Mär 25 10:12:13 f0.cockpit.lan ipa-server-guard[815]: Traceback (most recent call last): File "/usr/libexec/certmonger/ipa-server-guard", line 51, in <module> sys.exit(main()) File "/usr/libexec/certmonger/ipa-server-guard", line 45, in main sys.stderr.flush() File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 666, in __exit__ self.release() File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 672, in release self._do(self._release, owner) File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 713, in _do self._write(f) File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 751, in _write p.write(fileobj) File "/usr/lib64/python2.7/ConfigParser.py", line 407, in write fp.write("[%s]\n" % section) IOError: [Errno 28] No space left on device Mär 25 10:12:13 f0.cockpit.lan ipa-server-guard[804]: Traceback (most recent call last): File "/usr/libexec/certmonger/ipa-server-guard", line 51, in <module> sys.exit(main()) File "/usr/libexec/certmonger/ipa-server-guard", line 45, in main sys.stderr.flush() File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 666, in __exit__ self.release() File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 672, in release self._do(self._release, owner) File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 714, in _do return IOError: [Errno 28] No space left on device Mär 25 10:12:13 f0.cockpit.lan ipa-server-guard[819]: Traceback (most recent call last): File "/usr/libexec/certmonger/ipa-server-guard", line 51, in <module> sys.exit(main()) File "/usr/libexec/certmonger/ipa-server-guard", line 39, in main with certs.renewal_lock: File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 663, in __enter__ self.acquire() File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 669, in acquire self._do(self._acquire, owner) File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 713, in _do self._write(f) File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 751, in _write p.write(fileobj) File "/usr/lib64/python2.7/ConfigParser.py", line 407, in write fp.write("[%s]\n" % section) IOError: [Errno 28] No space left on device Mär 25 10:12:13 f0.cockpit.lan ipa-server-guard[820]: Traceback (most recent call last): File "/usr/libexec/certmonger/ipa-server-guard", line 51, in <module> sys.exit(main()) File "/usr/libexec/certmonger/ipa-server-guard", line 39, in main with certs.renewal_lock: File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 663, in __enter__ self.acquire() File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 669, in acquire self._do(self._acquire, owner) File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 713, in _do self._write(f) File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 751, in _write p.write(fileobj) File "/usr/lib64/python2.7/ConfigParser.py", line 407, in write fp.write("[%s]\n" % section) IOError: [Errno 28] No space left on device Mär 25 10:12:13 f0.cockpit.lan ipa-server-guard[822]: Traceback (most recent call last): File "/usr/libexec/certmonger/ipa-server-guard", line 51, in <module> sys.exit(main()) File "/usr/libexec/certmonger/ipa-server-guard", line 39, in main with certs.renewal_lock: File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 663, in __enter__ self.acquire() File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 669, in acquire self._do(self._acquire, owner) File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 713, in _do self._write(f) File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 751, in _write p.write(fileobj) File "/usr/lib64/python2.7/ConfigParser.py", line 407, in write fp.write("[%s]\n" % section) IOError: [Errno 28] No space left on device Mär 25 10:12:13 f0.cockpit.lan ipa-server-guard[821]: Traceback (most recent call last): File "/usr/libexec/certmonger/ipa-server-guard", line 51, in <module> sys.exit(main()) File "/usr/libexec/certmonger/ipa-server-guard", line 39, in main with certs.renewal_lock: File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 663, in __enter__ self.acquire() File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 669, in acquire self._do(self._acquire, owner) File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 713, in _do self._write(f) File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 751, in _write p.write(fileobj) File "/usr/lib64/python2.7/ConfigParser.py", line 407, in write fp.write("[%s]\n" % section) IOError: [Errno 28] No space left on device Mär 25 10:12:13 f0.cockpit.lan ipa-server-guard[807]: Traceback (most recent call last): File "/usr/libexec/certmonger/ipa-server-guard", line 51, in <module> sys.exit(main()) File "/usr/libexec/certmonger/ipa-server-guard", line 41, in main env=os.environ) File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 321, in run preexec_fn=preexec_fn) File "/usr/lib64/python2.7/subprocess.py", line 710, in __init__ errread, errwrite) File "/usr/lib64/python2.7/subprocess.py", line 1223, in _execute_child self.pid = os.fork() OSError: [Errno 12] Cannot allocate memory
Created attachment 1006237 [details] check-realms-testIpa-10.111.111.100-FAIL.journal.tgz Tarball of freeipa server journal
This breaks Cockpit development.
The interesting stuff is in /var/log/httpd/error_log. Anyway, it looks like your server has run out of space.
This has stopped happening. I will reopen when it happens again.