Created attachment 564458 [details]
db_seed.log from katello-configure run

After it boots, I try to connect to katello and it gives a 503 error.

I also see some SELinux errors:

type=AVC msg=audit(1329639011.078:46466): avc:  denied  { read } for  pid=4961 comm="httpd" name="webservices.wsgi" dev=dm-1 ino=163553 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:var_t:s0 tclass=file

type=AVC msg=audit(1329643386.918:46522): avc:  denied  { name_connect } for  pid=4970 comm="httpd" dest=5000 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:commplex_port_t:s0 tclass=tcp_socket

type=AVC msg=audit(1329644438.785:46603): avc:  denied  { execute } for  pid=7884 comm="httpd" path=2F746D702F6666696F3556714239202864656C6574656429 dev=dm-1 ino=65620 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:httpd_tmp_t:s0 tclass=file


type=AVC msg=audit(1329644439.114:46604): avc:  denied  { write } for  pid=7884 comm="httpd" name="events.log" dev=dm-1 ino=290039 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=system_u:object_r:httpd_log_t:s0 tclass=file


# audit2allow -al
#============= httpd_t ==============
#!!!! This avc can be allowed using the boolean 'httpd_can_network_connect'

allow httpd_t commplex_port_t:tcp_socket name_connect;
allow httpd_t var_t:file read;

Is there a way to recover from this issue?  When I reran katello-configure after boot, I kept getting this same error.  I removed katello (bug#767750) and reinstalled, but katello-configure still gives me the same error.

I also see this error when I run katello-configure from the command-line:
katello-configure 
Starting Katello configuration
The top-level log file is [/var/log/katello/katello-configure-20120219-050839/main.log]
err: /Stage[main]/Certs::Config/Exec[create-nss-db]/returns: change from notrun to 0 failed: /bin/rm -f /etc/pki/katello/nssdb//*; certutil -N -d '/etc/pki/katello/nssdb/' -f '/etc/katello/nss_db_password-file'; certutil -A -d '/etc/pki/katello/nssdb/' -n 'ca' -t 'TCu,Cu,Tuw' -a -i '/usr/share/katello/KATELLO-TRUSTED-SSL-CERT'; certutil -A -d '/etc/pki/katello/nssdb/' -n 'broker' -t ',,' -a -i '/etc/pki/tls/certs/qpid-broker.crt'; certutil -A -d '/etc/pki/katello/nssdb/' -n 'tomcat' -t ',,' -a -i '/etc/pki/tls/certs/httpd-ssl.crt' returned 255 instead of one of [0] at /usr/share/katello/install/puppet/modules/certs/manifests/config.pp:184

It looks like it is the final command that is failing:

certutil -A -d '/etc/pki/katello/nssdb/' -n 'tomcat' -t ',,' -a -i '/etc/pki/tls/certs/httpd-ssl.crt'

certutil:  unable to open "/etc/pki/tls/certs/httpd-ssl.crt" for reading (-5950, 2).

It looks like it stores things in /root/ssl-build/.  I cleaned that out and reran katello-configure and I get this error, which seems to match the original issue I was getting during kickstart:

err: /Stage[main]/Katello::Config/Exec[katello_seed_db]/returns: change from notrun to 0 failed: /usr/bin/env rake seed_with_logging --trace --verbose > /var/log/katello/katello-configure/db_seed.log 2>&1 && touch /var/lib/katello/db_seed_done returned 1 instead of one of [0] at /usr/share/katello/install/puppet/modules/katello/manifests/config.pp:183

/var/log/katello/katello-configure/db_seed.log shows the errors that I had originally:
Skipping index creation, cannot connect to ElasticSearch
(The original exception was: #<Errno::ECONNREFUSED: Connection refused - connect(2)>)

Argh, this was an issue of RAM.  I used an old virt-install script that only included 1 GB RAM.  I bumped it up to 2 GB and it works fine from the kickstart %post script now.

This might make a good kbase article.

Updated to 2012-02-22.1 code.

Still seeing errors running katello-configure in kickstart %post:

+ katello-configure
Starting Katello configuration
The top-level log file is [/var/log/katello/katello-configure-20120223-105217/main.log]
ESC[1;35merr: /Stage[main]/Postgres::Service/Service[postgresql]: Failed to call refresh: Could not restart Service[postgresql]: Execution of '/sbin/service postgresql restart' returned 1:  at /usr/share/katello/install/puppet/modules/postgres/manifests/service.pp:6ESC[0m
...

I subsequently have issues running commands to manage katello.  For instance, the first command I run after boot is:

katello -u admin -p admin provider create --org ACME_Corporation --name instructor
    ERROR:  duplicate key value violates unique constraint "providers_pkey"

When I run the same command a second time, it succeeds.  There was obviously an issue with the database when it ran in %post.

If I run katello-configure after the machine has booted (and not during kickstart), everything works fine.

For completeness, here are the other commands that consistently fail:

katello -u admin -p admin environment create --org ACME_Corporation --name dev --prior Library
    ERROR:  duplicate key value violates unique constraint
"environments_pkey"

The following command must be run thrice to complete successfully.  It gets two different errors on the first two runs:
katello -u admin -p admin org create --name Test_Org
    ERROR:  duplicate key value violates unique constraint
"organizations_pkey"
katello -u admin -p admin org create --name Test_Org
    Runtime Error Could not execute JDBC batch update at
org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleError:2,598
    (two separate errors)

katello -u admin -p admin user create --user bob --password bobby --email root@localhost
    ERROR:  duplicate key value violates unique constraint "users_pkey"

Just tested 2012-03-01.1 code drop.
katello-all-0.1.301-2.el6.noarch

katello-configure in %post of kickstart now runs without error.  However, I still get the duplicate key value violates unique constraint errors.  This may be related to the packages not installing correctly (bug#795602).

I installed katello-all 1.0.1 on RHEL 6.3 and this bug is still present. It seems that this bug can be random, because it succeeded one time and functioned normally.

(In reply to comment #11)
> I installed katello-all 1.0.1 on RHEL 6.3 and this bug is still present. It
> seems that this bug can be random, because it succeeded one time and
> functioned normally.

Comment#7 indicates this might be related to memory.  Can you describe how much memory is available on the system?

That VM had 1 or 2 GB RAM assigned. Unfortunately, I deleted it, but I have that VM backed up in state right before katello-configure, so it can be investigated further. I'm recovering it back now and I will try it with both 1 and 2 GB RAM.

(In reply to comment #13)
> That VM had 1 or 2 GB RAM assigned. Unfortunately, I deleted it, but I have
> that VM backed up in state right before katello-configure, so it can be
> investigated further. I'm recovering it back now and I will try it with both
> 1 and 2 GB RAM.

Thanks for hte info Milan.  If the problem reproduces with 1G of memory, I believe we can ignore the issue as 1G does not meet the minimum system requirements for System ENgine [1].  The same is true for a 2G system.  We may want to raise the alarms if the problem occurs on a system with 4G of memory.

[1] https://docs.redhat.com/docs/en-US/CloudForms/1.0/html/Installation_Guide/chap-Installation_Guide-Introduction.html#sect-Installation_Guide-Introduction-Prerequisites

mmccune: what's your take?  Is it acceptable to lean on the minimum system requirements (4G of memory) here?

So I tried to reproduce that bug with 1G and it successfully configured (slow as hell but it did well & everything works).

(In reply to comment #15)
> So I tried to reproduce that bug with 1G and it successfully configured
> (slow as hell but it did well & everything works).

Heh, thanks for trying to reproduce.  Such is the nature of memory contention issues.  There are *enumerable* factors that contribute to how much memory a running system uses.  I wouldn't be surprised if reproducing this issue was non-deterministic.  </$0.02>

Regarding comment #14, the prerequisites require a physical machine with a minimum of 4 GB RAM.  This is the Cloud Engine system.  There are no requirements that I have seen that state that the System Engine machine should have more than 1 GB RAM.  We really need to add some specs for the SE machine...

(In reply to comment #17)
> Regarding comment #14, the prerequisites require a physical machine with a
> minimum of 4 GB RAM.  This is the Cloud Engine system.  There are no
> requirements that I have seen that state that the System Engine machine
> should have more than 1 GB RAM.  We really need to add some specs for the SE
> machine...

FYI:

http://docs.redhat.com/docs/en-US/CloudForms/1.0/html/Installation_Guide/chap-Installation_Guide-Introduction.html

"You must meet the following conditions before installing CloudForms: "

This is for Cloudforms, it applies to both System Engine and Cloud Engine components of 'CloudForms'. This is my interpretation at least. 

The page then states it is recommended to install SE onto a guest OS instance running on the physical machine which runs CE. Hence, the 4 gig min, 8 recommended would be split in half for each OS as 2/2, or 4/4. 

I agree, docs could be clearer, but the above is my interpretation at least.

(RE: comment #18)
I agree with you Cliff.  With the 4 GB RAM minimum, System Engine should take 2 GB RAM, and our docs should state that.  However, comment # 14 dismissed the 2 GB case outright.

DOCS:

Lets split out the requirements to state that both CFSE and CFCE machines each need:

"64-bit architecture.
    Red Hat Enterprise Linux 6.2 or newer.
    At least 4GB and ideally 8GB of memory. It is also recommended to use swap space where possible. 

make it clear that each instance (CE and SE) needs those things

Hi All,
This information has been updated in the System Engine User guide. Please review this section:
http://documentation-stage-02.lab.eng.bne.redhat.com/docs/en-US/CloudForms/1.1/html/Installation_Guide/Prerequisites.html

Regards,
Shikha

Correction:
The update has occured in System Engine Installation Guide instead of System Engine User Guide.

http://documentation-stage-02.lab.eng.bne.redhat.com/docs/en-US/CloudForms/1.1/html/Installation_Guide/Prerequisites.html

regards,
shikha

This documentation has now been dropped to translation ahead of publication. For any further issues, please open a new a bug.

LKB

This document is now publicly available on access.redhat.com. For any further issues, please raise a new bug.

LKB