Red Hat Bugzilla – Bug 605743
No errors on why post failed with snippets.
Last modified: 2013-08-15 04:36:54 EDT
Description of problem:
Post scripts fail if there is any kind of error in a snippet. Sadly, there is no error log of why they failed. This makes troubleshooting a complete nightmare and some customers are even afraid to use snippets do to this failure.
Version-Release number of selected component (if applicable):
Create a snippet with a missing "'" or incorrect configuration. For example "sed -i /eth0.//' /etc/sysconfig/network will cause post to fail. There is no error about why. It just simply won't run anything in post.
Steps to Reproduce:
1. mess up a snippet slightly, but not enough for the template error to come up.
2. Kickstart a system.
3. Wonder why none of the 5 or 6 post snippets worked.
Unknown failure and then you have to kickstart a machine with one snippet at a time until you find the bad snippet, then you have to look at every line.
An error message. A little bit of the "error on line #" Or maybe at least, error with snippet('spacewalk/1/X')
It would also be nice if there was a way to have cheetah or anaconda test the snippets so we don't have to install a machine (15 minutes) then check and see if the post worked.
*** Bug 640106 has been marked as a duplicate of this bug. ***
I agree, it's pretty irritating when that happens. Getting this features would save so much time troubleshooting and I haven't yet heard of someone that hasn't spend hours troubleshooting errors like this.
Ofcourse, one can try and run a snippet locally on a machine first, after each and every change.. or if you hit such an issue start the first snippet with:
while true; do sleep 1; done
..and then change tty to run each snippet manually.
..but that is just as fun as it sounds.
It should be noted that this goes for any part of the kickstart, pre, post nochroot or post.
Using "sh -n" and a fancy cronjob that runs once a minute I was with ~150 LoC able to implement an automatic syntax check for kickstart snippets.
If a snippets is changed (you can see that by diffing against the files in /var/lib/rhn/kickstarts/snippets/1/) then I run "sh -n on /var/lib/rhn/kickstarts/snippets/1/my-post-snippet" and get an e-mail like below:
From: root <email@example.com>
Date: March 26, 2012 2:09:02 PM GMT+02:00
To: Magnus Glantz <my-email@address>
Subject: Syntax error in kickstart snippet: dev-linuxts-post-main: Mon Mar 26 12:09:02 UTC 2012
/var/lib/rhn/kickstarts/snippets/1/dev-linuxts-post-main: line 72: syntax error near unexpected token `('
/var/lib/rhn/kickstarts/snippets/1/dev-linuxts-post-main: line 72: ` echo "Setting NTP server to $nsbserver ($reversensb)"'
I'll see what I can do to share the code (written in company time..).
Not sure how to syntax check Python or other languages, "sh -n" doesn't work with Python.
The ability to log errors already exists today.
Go to the kickstart profile details page and check the box next to "Log custom post scripts." If I do that and then run a kickstart with a 1-line script that is the sed command from comment 0, I end up with this in /root/ks-post-log.1:
/tmp/ks-script-Vd6VzT: line1: unexpected EOF while looking for matching `''
/tmp/ks-script-Vd6VzT: line2: syntax error: unexpected end of file
The logging feature may not be in the most intuitive place since it's on the kickstart details page and not enableable for individual scripts, but it is present.
I will close this bug as current release.
You may also find the errorOnFail option that will be selectable for kickstart scripts in Sat 5.6 useful. You can also play around with getting an interactive terminal during a pre / post script in order to really thoroughly debug your scripts:
(the "text" option is settable at the Kickstart Details -> Advanced Options page in Satellite)
I see now on a more careful reading of the bug that there are actually two competing requests here.
One is the "there's no way to get an error log from my scripts" bug that is incorrect and that I addressed with comment 20.
The other is a "it would be really nice if we could have syntax checking on scripts in Satellite without having to run an actual kickstart" idea.
I believe the two should be separated. I consider this bug to be about the former since that was the main thrust of the original problem. The latter is a fine idea and in fact it would be really nice, but I think it should be filed as a separate RFE and tracked / prioritized appropriately.
I hope this clears up some confusion.
I've asked Anders Reyner-Karlsson to create a separeate RFE for the syntax check part.
As a note, using diff to detect when a snippet is changed and "file" to detect what a kickstart snippet runs for kind of stuff in combination with..
sh -n /var/lib/rhn/kickstarts/snippets/1/$FILE
python -c "import py_compile; py_compile.compile(r'/var/lib/rhn/kickstarts/snippets/1/$FILE')"
..depending on what's detected, we've now got a working syntax check for kickstart snippets.
Sorry, Anders Rayner-Karlsson (of Red Hat).