Bug 1120818

Summary: RHEL 7 kickstart trees don't appear in web UI
Product: Red Hat Satellite 5 Reporter: Jason Russler <jrussler>
Component: WebUIAssignee: Stephen Herr <sherr>
Status: CLOSED NOTABUG QA Contact: Red Hat Satellite QA List <satqe-list>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 560CC: cperry, cstlaure, mmello
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-07-21 18:47:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 465198, 924189    

Description Jason Russler 2014-07-17 18:51:07 UTC
Description of problem:

Synced Red Hat Enterprise Linux Server 7 channel (rhel-x86_64-server-7) complete with kickstart distribution tree. While "cobbler distro list" shows the installed tree and kickstarts can be created and used to successfuly PXE install hosts through cobbler, the web UI does not show the tree and the kickstart profile creation wizard can't be used when using RHEL 7 as the base channel. The kickstart trees do now show up in the UI.

If I manually attempt to create the tree, the web interface presents the following error:

"The initrd could not be found at the specified location: /var/satellite/rhn/kickstart/ks-rhel-x86_64-server-7-7.0/images/pxeboot/initrd.img"

Version-Release number of selected component (if applicable):

Ssatellite 5.6 (on RHEL 6.5 x86_64)

How reproducible:

This problem is apparent on a staging Satellite and RHEL 7 channels synced to the consumers exihibit the same behavior. I have yet to blow away the channel and resync.

Steps to Reproduce:
1. rhel-x86_64-server-7
2. wait for taskomatic jobs to run or run them manually
3. attempt to create a kickstart profile, select "Red Hat Enterprise Linux Server (v. 7 for 64-bit x86_64)" as the base channel

Actual results:

No kickstartable tree is available and the creation of a profile cannot continue.

Expected results:

Available kickstart tree so that a kickstart profile can be generated.


Additional info:

The path I provide to the UI when manually trying to create a kickstart tree is 100% good. The error even spells out the correct path to the initrd.img file. SElinux is not enforcing on this system and the perms for the rhel7 tree are the same as the perms for the rhel6 trees that do work. Something about how Satellite validates the tree is not working for RHEL7? taskomatic logs aren't revealing anything alarming (to me anyway).

Comment 1 Jason Russler 2014-07-17 18:53:20 UTC
Bad typo in the description, should read "The kickstart trees do _not_ show up in the UI."

Comment 2 Stephen Herr 2014-07-17 20:19:22 UTC
Hi Jason,

Is there perhaps anything of interest in /var/log/tomcat6/catalina.out?

What versions of spacewalk-java and spacewalk-taskomatic do you have installed?

Comment 3 Stephen Herr 2014-07-18 12:36:21 UTC
I have not been able to reproduce this so far. I've synced the rhel-x86_64-server-7 channel on both a fully-updated Spacewalk maching and a fully-updated Satellite 5.6 machine, and in both cases the kickstart distribution is available when creating profiles.

Jason, in addition to my questions in comment 2, could you please take another look at the permissions of the kickstart trees? Ensure that regardless of who ownes the files that the apache user has both read and execute permissions on every directory leading up to the initrd.img / vmlinuz files, and has read to those files themselves. Thanks.

Comment 4 Jason Russler 2014-07-18 14:32:28 UTC
Stephen, /var/log/tomcat6/catalina.out does show some errors (bottom) but I can't tell if they're related or not. Paths to kernel and ramdisk from apache's perspective seem ok (no selinux on these hosts):

--
# sudo -u apache file /var/satellite/rhn/kickstart/ks-rhel-x86_64-server-7-7.0/images/pxeboot/initrd.img
/var/satellite/rhn/kickstart/ks-rhel-x86_64-server-7-7.0/images/pxeboot/initrd.img: xz compressed data
# sudo -u apache file //var/satellite/rhn/kickstart/ks-rhel-x86_64-server-7-7.0/images/pxeboot/vmlinuz
//var/satellite/rhn/kickstart/ks-rhel-x86_64-server-7-7.0/images/pxeboot/vmlinuz: Linux kernel x86 boot executable bzImage, version 3.10.0-123.el7.x86_64 (mockbuil, RO-rootFS, swap_dev 0x4, Normal VGA
--

Versions:
spacewalk-java: 2.0.2-79.el6sat
spacewalk-taskomatic: 2.0.2-79.el6sat

It appears there may be a broader issue. Attempts to sync the RHEL7 channels from this staging Satellite out to the slaves (5.6) are failing. "satellite-sync -c rhel-x86_64-server-7" on a slave pulls the channel and tree with no errors but the channel is not showing up in the web UI at all - though cobbler still sees the distro (which is useable). I can't remove the channel from the slave and resync because I can't see it through the UI at all - though syncing happily updates the channel like it's there.

All of the RHEL6 channels are still sycing and functioning.

Should I clear the RHEL 7 base channel on the master and resync it from base content ISOs or is there more you want to see from the current state? These channels are not in use so we can experiment if you want.


/var/log/tomcat6/catalina.out:
--
2014-07-14 10:36:02,939 [TP-Processor8] ERROR com.redhat.rhn.frontend.action.common.DownloadFile - /var/satellite/rhn/kickstart/ks-rhel-x86_64-server-6-6.5/images/updates.img Not Found .. 404!
2014-07-14 10:36:26,530 [TP-Processor19] ERROR com.redhat.rhn.frontend.action.common.DownloadFile - /var/cache/rhn/repodata/.treeinfo Not Found .. 404!
2014-07-14 10:36:26,548 [TP-Processor13] ERROR com.redhat.rhn.frontend.action.common.DownloadFile - /var/cache/rhn/repodata/treeinfo Not Found .. 404!
2014-07-15 09:22:21,258 [TP-Processor18] ERROR com.redhat.rhn.common.localization.LocalizationService - *** ERROR: Message with id: [deploy.jsp.selection] not found.*** called by com.redhat.rhn.frontend.struts.XmlMessageResources.getMessage(XmlMessageResources.java:85)
2014-07-15 09:22:21,258 [TP-Processor18] ERROR com.redhat.rhn.common.localization.LocalizationService - *** ERROR: Message with id: [deploy.jsp.selection] not found.*** called by com.redhat.rhn.frontend.struts.XmlMessageResources.getMessage(XmlMessageResources.java:85)
2014-07-15 09:33:11,607 [TP-Processor22] WARN  com.redhat.rhn.common.hibernate.EmptyVarcharInterceptor - Object com.redhat.rhn.domain.server.Location is setting empty string rack
2014-07-15 09:33:11,611 [TP-Processor22] WARN  com.redhat.rhn.common.hibernate.EmptyVarcharInterceptor - Object com.redhat.rhn.domain.server.Location is setting empty string room
2014-07-15 09:33:11,611 [TP-Processor22] WARN  com.redhat.rhn.common.hibernate.EmptyVarcharInterceptor - Object com.redhat.rhn.domain.server.Location is setting empty string building
2014-07-15 09:33:11,611 [TP-Processor22] WARN  com.redhat.rhn.common.hibernate.EmptyVarcharInterceptor - Object com.redhat.rhn.domain.server.Location is setting empty string address1
2014-07-15 09:33:11,611 [TP-Processor22] WARN  com.redhat.rhn.common.hibernate.EmptyVarcharInterceptor - Object com.redhat.rhn.domain.server.Location is setting empty string address2
2014-07-15 09:33:11,611 [TP-Processor22] WARN  com.redhat.rhn.common.hibernate.EmptyVarcharInterceptor - Object com.redhat.rhn.domain.server.Location is setting empty string city
2014-07-15 09:33:11,611 [TP-Processor22] WARN  com.redhat.rhn.common.hibernate.EmptyVarcharInterceptor - Object com.redhat.rhn.domain.server.Location is setting empty string state
2014-07-15 09:33:11,611 [TP-Processor22] WARN  com.redhat.rhn.common.hibernate.EmptyVarcharInterceptor - Object com.redhat.rhn.domain.server.Location is setting empty string country
2014-07-15 10:23:47,238 [TP-Processor12] ERROR com.redhat.rhn.frontend.action.common.DownloadFile - /var/satellite/rhn/kickstart/ks-rhel-x86_64-server-6-6.5/images/updates.img Not Found .. 404!
2014-07-15 10:24:19,925 [TP-Processor8] ERROR com.redhat.rhn.frontend.action.common.DownloadFile - /var/cache/rhn/repodata/.treeinfo Not Found .. 404!
2014-07-15 10:24:19,935 [TP-Processor18] ERROR com.redhat.rhn.frontend.action.common.DownloadFile - /var/cache/rhn/repodata/treeinfo Not Found .. 404!
2014-07-15 11:32:46,977 [TP-Processor15] ERROR com.redhat.rhn.frontend.action.common.DownloadFile - /var/satellite/rhn/kickstart/ks-rhel-x86_64-server-6-6.5/images/updates.img Not Found .. 404!
2014-07-15 11:33:05,487 [TP-Processor19] ERROR com.redhat.rhn.frontend.action.common.DownloadFile - /var/cache/rhn/repodata/.treeinfo Not Found .. 404!
2014-07-15 11:33:05,497 [TP-Processor20] ERROR com.redhat.rhn.frontend.action.common.DownloadFile - /var/cache/rhn/repodata/treeinfo Not Found .. 404!
2014-07-15 11:35:14,044 [TP-Processor12] ERROR com.redhat.rhn.frontend.action.common.DownloadFile - /var/satellite/rhn/kickstart/ks-rhel-x86_64-server-6-6.5/images/updates.img Not Found .. 404!
2014-07-15 11:35:55,534 [TP-Processor3] ERROR com.redhat.rhn.frontend.action.common.DownloadFile - /var/cache/rhn/repodata/.treeinfo Not Found .. 404!
2014-07-15 11:35:55,542 [TP-Processor17] ERROR com.redhat.rhn.frontend.action.common.DownloadFile - /var/cache/rhn/repodata/treeinfo Not Found .. 404!
2014-07-16 14:13:28,495 [TP-Processor10] ERROR com.redhat.rhn.frontend.action.common.DownloadFile - /var/satellite/rhn/kickstart/ks-rhel-x86_64-server-6-6.5/images/updates.img Not Found .. 404!
2014-07-16 14:14:31,226 [TP-Processor15] ERROR com.redhat.rhn.frontend.action.common.DownloadFile - /var/cache/rhn/repodata/.treeinfo Not Found .. 404!
2014-07-16 14:14:31,237 [TP-Processor2] ERROR com.redhat.rhn.frontend.action.common.DownloadFile - /var/cache/rhn/repodata/treeinfo Not Found .. 404!
--

Comment 5 Stephen Herr 2014-07-18 15:08:12 UTC
Can you please try doing a 'rhn-satellite restart' and see if that fixes the UI visibility problem?

Comment 6 Jason Russler 2014-07-18 15:14:36 UTC
That was tried earlier this week. No good.

Comment 7 Stephen Herr 2014-07-18 15:27:34 UTC
I don't know what else to suggest then. Since it's not a reproducable problem and you're the only one that's encountered it there must be something special about your system that is causing this to happen, but I'm out of ideas as to where to look for it.

The 404s are nothing to be concerned about, that's a normal part of the kickstart process. Anaconda guesses at urls until it finds what it's looking for.

The fact that the kickstart distribution appears in cobbler means that:
1) the satellite-sync finished successfully and the Satellite db was correctly updated.
2) the CobblerSync task in Taskomitic completed successfully and synced info over to cobbler

I had thought it must be a case of tomcat's hibernate having a bad cache, but restarting Satellite should have resolved the issue if that was the case. You can experiment with deleting the channel and re-syncing if you want, it's worth a try.

Comment 8 Stephen Herr 2014-07-21 18:40:23 UTC
Jason reports in an email that the problem was that the /var/satellite/rhn/kickstart/ks-rhel-x86_64-server-7-7.0/images had 700 permissions instead of 755.

This raises an important addendum to comment 3, both the apache *and* tomcat users must have read / execute permissions on the directories leading up to initrd.img and vmlinuz, and read on the files themselves.

I do not know how your Satellite had the wrong permissions set on that directory, I see no way for that to happen through the code and I cannot reproduce the behavior in my testing. Perhaps the directory existed before the satellite-sync on your system for some reason?

Comment 9 Stephen Herr 2014-07-21 18:47:40 UTC
Jason reports in an email that the problematic system had been security hardened and a umask had been set that disallowed satellite-sync from setting the correct 755 permissions on these directories.

Comment 10 Clifford Perry 2014-08-08 19:35:08 UTC
*** Bug 1086352 has been marked as a duplicate of this bug. ***