Bug 1205593

Summary: Intermittent ipa/json "Internal Server Error"
Product: [Fedora] Fedora Reporter: Stef Walter <stefw>
Component: freeipaAssignee: IPA Maintainers <ipa-maint>
Status: CLOSED INSUFFICIENT_DATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 22CC: abokovoy, dperpeet, ipa-maint, jcholast, jscotka, mkosek, mvollmer, pvoborni, rcritten, ssorce, stefw
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-03-31 06:13:01 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1204627    
Attachments:
Description Flags
check-realms-testIpa-10.111.111.100-FAIL.journal.tgz none

Description Stef Walter 2015-03-25 09:33:03 UTC
Description of problem:

Intermittent failure to enroll a system, due to "Internal Server Error" returned from https://f0.cockpit.lan/ipa/json

Version-Release number of selected component (if applicable):

freeipa-client-4.1.3-3.fc22.x86_64
freeipa-server-4.1.3-2.fc21.x86_64

How reproducible:

On my machine 1 in 5 runs of the Cockpit CI system.

Actual results:

Mär 25 10:12:10 Hostname: x0.cockpit.lan
Mär 25 10:12:10 Realm: COCKPIT.LAN
Mär 25 10:12:10 DNS Domain: cockpit.lan
Mär 25 10:12:10 IPA Server: f0.cockpit.lan
Mär 25 10:12:10 BaseDN: dc=cockpit,dc=lan
Mär 25 10:12:10 Synchronizing time with KDC...
Mär 25 10:12:10 Attempting to sync time using ntpd.  Will timeout after 15 seconds
Mär 25 10:12:11 Successfully retrieved CA cert
Mär 25 10:12:11     Subject:     CN=Certificate Authority,O=COCKPIT.LAN
Mär 25 10:12:11     Issuer:      CN=Certificate Authority,O=COCKPIT.LAN
Mär 25 10:12:11     Valid From:  Mon Mar 09 11:06:52 2015 UTC
Mär 25 10:12:11     Valid Until: Fri Mar 09 11:06:52 2035 UTC
Mär 25 10:12:11 
Mär 25 10:12:13 Enrolled in IPA realm COCKPIT.LAN
Mär 25 10:12:13 Created /etc/ipa/default.conf
Mär 25 10:12:14 New SSSD config will be created
Mär 25 10:12:14 Configured sudoers in /etc/nsswitch.conf
Mär 25 10:12:14 Configured /etc/sssd/sssd.conf
Mär 25 10:12:14 Configured /etc/krb5.conf for IPA realm COCKPIT.LAN
Mär 25 10:12:14 trying https://f0.cockpit.lan/ipa/json
Mär 25 10:12:14 Forwarding 'ping' to json server 'https://f0.cockpit.lan/ipa/json'
Mär 25 10:12:14 Cannot connect to the server due to generic error: cannot connect to 'https://f0.cockpit.lan/ipa/json': Internal Server Error
Mär 25 10:12:14 Installation failed. Rolling back changes.
Mär 25 10:12:15 Failed to list certificates in /etc/ipa/nssdb: Command ''/usr/bin/certutil' '-d' '/etc/ipa/nssdb' '-L'' returned non-zero exit status 255
Mär 25 10:12:15 Failed to remove /etc/ipa/nssdb/cert8.db: [Errno 2] No such file or directory: '/etc/ipa/nssdb/cert8.db'
Mär 25 10:12:15 Failed to remove /etc/ipa/nssdb/key3.db: [Errno 2] No such file or directory: '/etc/ipa/nssdb/key3.db'
Mär 25 10:12:15 Failed to remove /etc/ipa/nssdb/secmod.db: [Errno 2] No such file or directory: '/etc/ipa/nssdb/secmod.db'
Mär 25 10:12:15 Failed to remove /etc/ipa/nssdb/pwdfile.txt: [Errno 2] No such file or directory: '/etc/ipa/nssdb/pwdfile.txt'
Mär 25 10:12:15 Unenrolling client from IPA server
Mär 25 10:12:16 Removing Kerberos service principals from /etc/krb5.keytab
Mär 25 10:12:16 Disabling client Kerberos and LDAP configurations
Mär 25 10:12:16 Redundant SSSD configuration file /etc/sssd/sssd.conf was moved to /etc/sssd/sssd.conf.deleted
Mär 25 10:12:16 Restoring client configuration files
Mär 25 10:12:16 nscd daemon is not installed, skip configuration
Mär 25 10:12:16 nslcd daemon is not installed, skip configuration
Mär 25 10:12:16 
Mär 25 10:12:16 Client uninstall complete.

Expected results:

Machine joined successfully.

Comment 1 Stef Walter 2015-03-25 09:35:12 UTC
Messages from the IPA server journal:

-- Logs begin at Mo 2015-03-09 11:57:56 CET, end at Mi 2015-03-25 10:14:10 CET. --
Mär 25 10:12:13 f0.cockpit.lan ipa-server-guard[803]: Traceback (most recent call last):
                                                         File "/usr/libexec/certmonger/ipa-server-guard", line 51, in <module>
                                                           sys.exit(main())
                                                         File "/usr/libexec/certmonger/ipa-server-guard", line 41, in main
                                                           env=os.environ)
                                                         File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 321, in run
                                                           preexec_fn=preexec_fn)
                                                         File "/usr/lib64/python2.7/subprocess.py", line 710, in __init__
                                                           errread, errwrite)
                                                         File "/usr/lib64/python2.7/subprocess.py", line 1223, in _execute_child
                                                           self.pid = os.fork()
                                                       OSError: [Errno 12] Cannot allocate memory
Mär 25 10:12:13 f0.cockpit.lan ipa-server-guard[815]: Traceback (most recent call last):
                                                         File "/usr/libexec/certmonger/ipa-server-guard", line 51, in <module>
                                                           sys.exit(main())
                                                         File "/usr/libexec/certmonger/ipa-server-guard", line 45, in main
                                                           sys.stderr.flush()
                                                         File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 666, in __exit__
                                                           self.release()
                                                         File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 672, in release
                                                           self._do(self._release, owner)
                                                         File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 713, in _do
                                                           self._write(f)
                                                         File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 751, in _write
                                                           p.write(fileobj)
                                                         File "/usr/lib64/python2.7/ConfigParser.py", line 407, in write
                                                           fp.write("[%s]\n" % section)
                                                       IOError: [Errno 28] No space left on device
Mär 25 10:12:13 f0.cockpit.lan ipa-server-guard[804]: Traceback (most recent call last):
                                                         File "/usr/libexec/certmonger/ipa-server-guard", line 51, in <module>
                                                           sys.exit(main())
                                                         File "/usr/libexec/certmonger/ipa-server-guard", line 45, in main
                                                           sys.stderr.flush()
                                                         File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 666, in __exit__
                                                           self.release()
                                                         File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 672, in release
                                                           self._do(self._release, owner)
                                                         File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 714, in _do
                                                           return
                                                       IOError: [Errno 28] No space left on device
Mär 25 10:12:13 f0.cockpit.lan ipa-server-guard[819]: Traceback (most recent call last):
                                                         File "/usr/libexec/certmonger/ipa-server-guard", line 51, in <module>
                                                           sys.exit(main())
                                                         File "/usr/libexec/certmonger/ipa-server-guard", line 39, in main
                                                           with certs.renewal_lock:
                                                         File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 663, in __enter__
                                                           self.acquire()
                                                         File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 669, in acquire
                                                           self._do(self._acquire, owner)
                                                         File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 713, in _do
                                                           self._write(f)
                                                         File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 751, in _write
                                                           p.write(fileobj)
                                                         File "/usr/lib64/python2.7/ConfigParser.py", line 407, in write
                                                           fp.write("[%s]\n" % section)
                                                       IOError: [Errno 28] No space left on device
Mär 25 10:12:13 f0.cockpit.lan ipa-server-guard[820]: Traceback (most recent call last):
                                                         File "/usr/libexec/certmonger/ipa-server-guard", line 51, in <module>
                                                           sys.exit(main())
                                                         File "/usr/libexec/certmonger/ipa-server-guard", line 39, in main
                                                           with certs.renewal_lock:
                                                         File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 663, in __enter__
                                                           self.acquire()
                                                         File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 669, in acquire
                                                           self._do(self._acquire, owner)
                                                         File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 713, in _do
                                                           self._write(f)
                                                         File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 751, in _write
                                                           p.write(fileobj)
                                                         File "/usr/lib64/python2.7/ConfigParser.py", line 407, in write
                                                           fp.write("[%s]\n" % section)
                                                       IOError: [Errno 28] No space left on device
Mär 25 10:12:13 f0.cockpit.lan ipa-server-guard[822]: Traceback (most recent call last):
                                                         File "/usr/libexec/certmonger/ipa-server-guard", line 51, in <module>
                                                           sys.exit(main())
                                                         File "/usr/libexec/certmonger/ipa-server-guard", line 39, in main
                                                           with certs.renewal_lock:
                                                         File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 663, in __enter__
                                                           self.acquire()
                                                         File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 669, in acquire
                                                           self._do(self._acquire, owner)
                                                         File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 713, in _do
                                                           self._write(f)
                                                         File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 751, in _write
                                                           p.write(fileobj)
                                                         File "/usr/lib64/python2.7/ConfigParser.py", line 407, in write
                                                           fp.write("[%s]\n" % section)
                                                       IOError: [Errno 28] No space left on device
Mär 25 10:12:13 f0.cockpit.lan ipa-server-guard[821]: Traceback (most recent call last):
                                                         File "/usr/libexec/certmonger/ipa-server-guard", line 51, in <module>
                                                           sys.exit(main())
                                                         File "/usr/libexec/certmonger/ipa-server-guard", line 39, in main
                                                           with certs.renewal_lock:
                                                         File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 663, in __enter__
                                                           self.acquire()
                                                         File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 669, in acquire
                                                           self._do(self._acquire, owner)
                                                         File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 713, in _do
                                                           self._write(f)
                                                         File "/usr/lib/python2.7/site-packages/ipaserver/install/certs.py", line 751, in _write
                                                           p.write(fileobj)
                                                         File "/usr/lib64/python2.7/ConfigParser.py", line 407, in write
                                                           fp.write("[%s]\n" % section)
                                                       IOError: [Errno 28] No space left on device
Mär 25 10:12:13 f0.cockpit.lan ipa-server-guard[807]: Traceback (most recent call last):
                                                         File "/usr/libexec/certmonger/ipa-server-guard", line 51, in <module>
                                                           sys.exit(main())
                                                         File "/usr/libexec/certmonger/ipa-server-guard", line 41, in main
                                                           env=os.environ)
                                                         File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 321, in run
                                                           preexec_fn=preexec_fn)
                                                         File "/usr/lib64/python2.7/subprocess.py", line 710, in __init__
                                                           errread, errwrite)
                                                         File "/usr/lib64/python2.7/subprocess.py", line 1223, in _execute_child
                                                           self.pid = os.fork()
                                                       OSError: [Errno 12] Cannot allocate memory

Comment 2 Stef Walter 2015-03-25 09:37:22 UTC
Created attachment 1006237 [details]
check-realms-testIpa-10.111.111.100-FAIL.journal.tgz

Tarball of freeipa server journal

Comment 3 Stef Walter 2015-03-25 09:38:36 UTC
This breaks Cockpit development.

Comment 4 Jan Cholasta 2015-03-25 11:00:45 UTC
The interesting stuff is in /var/log/httpd/error_log.

Anyway, it looks like your server has run out of space.

Comment 5 Stef Walter 2015-03-31 06:13:01 UTC
This has stopped happening. I will reopen when it happens again.