Bug 889329

+++ This bug was initially created as a clone of Bug #886709 +++

At the end of bootstrap, fetch log from host into /var/log/ovirt-engine/host-deploy.

This eases problem determination process as all logs reside on engine side.

--- Additional comment from Alon Bar-Lev on 2012-12-12 18:18:48 EST ---

In most of the cases logs will be collected.

If severe (protocol violation, communication lost) error is detected, then collection is not available, and log file is available at host machine.

--- Additional comment from Simon Grinberg on 2012-12-13 13:17:05 EST ---

On on hand it looks like an improvements, since now all the logs are being collected in a central location (except for the host logs collected by the log collector tool)

However the collection instructions become more complex to the end user or GSS.

Before this change is was always: 
1. Go to host:/tmp/xxx.log
Now it's 
1. Go to engine-host:/var/log/ovirt-engine/host-deploy and if it's not there then,
2. Go to host:/tmp/xxx.log

Is that correct?

--- Additional comment from Alon Bar-Lev on 2012-12-13 13:26:08 EST ---

Unless something terrible wrong logs will be at engine. Much easier.

Also, in fedora and others, /tmp/ is cleared after boot... so you don't have anything.

During deployment you get a message were logs is at host and where they are stored at engine. So you know where to look.

Example of events of new implementation:

---

2012-Dec-13, 20:13 Host test1 installed
2012-Dec-13, 20:13 Installing Host test1. Stage: Termination.
2012-Dec-13, 20:13 Installing Host test1. Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20121213201308-10.35.0.73.log'.
2012-Dec-13, 20:13 Installing Host test1. Reboot scheduled.	
2012-Dec-13, 20:13 Installing Host test1. Stage: Pre-termination.
2012-Dec-13, 20:13 Installing Host test1. Reboot scheduled.
2012-Dec-13, 20:13 Installing Host test1. Stage: Closing up.
2012-Dec-13, 20:13 Installing Host test1. Stage: Transaction commit.
2012-Dec-13, 20:13 Installing Host test1. Enrolling certificate.
2012-Dec-13, 20:13 Installing Host test1. Setting time.
2012-Dec-13, 20:13 Installing Host test1. Stage: Misc configuration.
2012-Dec-13, 20:13 Installing Host test1. Stage: Package installation.
2012-Dec-13, 20:13 Installing Host test1. Stage: Transaction setup.
2012-Dec-13, 20:13 Installing Host test1. Hardware supports virtualization.
2012-Dec-13, 20:13 Installing Host test1. Stage: Setup validation.
2012-Dec-13, 20:13 Installing Host test1. Enfocing host reboot.
2012-Dec-13, 20:13 Installing Host test1. Logs at host located at: '/tmp/ovirt-host-deploy-20121213201256.log'.
2012-Dec-13, 20:13 Installing Host test1. Stage: Environment customization.
2012-Dec-13, 20:13 Installing Host test1. Stage: Programs detection.
2012-Dec-13, 20:12 Installing Host test1. Stage: Installation packages setup.
2012-Dec-13, 20:12 Installing Host test1. Stage: Environment setup.
2012-Dec-13, 20:12 Installing Host test1. Stage: Initializing.
2012-Dec-13, 20:12 Installing Host test1. Connected to host 10.35.0.73 with SSH key fingerprint: 74:b2:98:dd:b4:31:e3:ef:48:71:90:aa:71:d4:f1:db.

--- Additional comment from Simon Grinberg on 2012-12-19 07:59:59 EST ---

(In reply to comment #3)

> 
> During deployment you get a message were logs is at host and where they are
> stored at engine. So you know where to look.
> 
> 2012-Dec-13, 20:13 Host test1 installed
> 2012-Dec-13, 20:13 Installing Host test1. Stage: Termination.
> 2012-Dec-13, 20:13 Installing Host test1. Retrieving installation logs to:
> '/var/log/ovirt-engine/host-deploy/ovirt-20121213201308-10.35.0.73.log'.

Good

> Also, in fedora and others, /tmp/ is cleared after boot... so you don't have anything.

2012-Dec-13, 20:13 Installing Host test1. Logs at host located at: '/tmp/ovirt-host-deploy-20121213201256.log'. 

Not good, since many times if you can't retrieve the logs then you may have to reboot the machine to retrieve network connection (This is the first thing a remote user (without remote console) will do before actually going to the server room to get console access) 

If /tmp is deleted we may need to consider a different location like /var/log

--- Additional comment from Alon Bar-Lev on 2012-12-19 08:50:19 EST ---

(In reply to comment #4)
> (In reply to comment #3)

> 2012-Dec-13, 20:13 Installing Host test1. Logs at host located at:
> '/tmp/ovirt-host-deploy-20121213201256.log'. 
> 
> Not good, since many times if you can't retrieve the logs then you may have
> to reboot the machine to retrieve network connection (This is the first
> thing a remote user (without remote console) will do before actually going
> to the server room to get console access) 
> 
> If /tmp is deleted we may need to consider a different location like /var/log

/var/log is managed directory, you cannot put there logs of junk.

managed directory = each file has owner (package).
managed directory = each file is monitored and cleaned up (logrotate and similar).

The deployment process is not installed on the machine, hence should not write to /var/log.

/tmp is the right place, as done in the past.

sysadmin can override this location using /etc/ovirt-host-config.conf, if indeed there is a problem during deployment, host reboot but engine cannot fetch log.

--- Additional comment from Stephen Gordon on 2012-12-19 16:57:16 EST ---

(In reply to comment #4)
> If /tmp is deleted we may need to consider a different location like /var/log

Note that in Fedora 18 after much back and forth the default will be for /tmp to be a tmpfs filesystem. That means it is wiped on reboot. I would expect this to impact upstream ovirt and potentially future versions of RHEV (if this change ever makes it into RHEL).

--- Additional comment from Alon Bar-Lev on 2012-12-19 16:59:48 EST ---

(In reply to comment #6)
> (In reply to comment #4)
> > If /tmp is deleted we may need to consider a different location like /var/log
> 
> Note that in Fedora 18 after much back and forth the default will be for
> /tmp to be a tmpfs filesystem. That means it is wiped on reboot. I would
> expect this to impact upstream ovirt and potentially future versions of RHEV
> (if this change ever makes it into RHEL).

Known issue, refer to previous comments.

I am not sure it is tmpfs, but sure it is cleared on startup, many distributions already took this approach which is correct.

Thanks!

--- Additional comment from Stephen Gordon on 2012-12-19 17:06:33 EST ---

Right, but the proposed resolution to the scenario Simon put forward looks like being to push a very unintuitive procedure off to documentation (otherwise how would the admin know to tweak the setting in /etc/ovirt-host-config.conf)?

What is the reason for preferring /tmp here over /var/tmp (at least for upstream) which is persisted?

--- Additional comment from Alon Bar-Lev on 2012-12-19 17:12:19 EST ---

(In reply to comment #8)
> Right, but the proposed resolution to the scenario Simon put forward looks
> like being to push a very unintuitive procedure off to documentation
> (otherwise how would the admin know to tweak the setting in
> /etc/ovirt-host-config.conf)?

The documentation should state that the logs are collected to the engine machine at /var/log/ovirt-engine/host-deploy.

In cause of severe installation failure, the engine will not be able to collect the logs, and the host will not reboot (as there is installation failure), in this case there is a fallback to consult logs at /tmp of the host machine.

In case there is a severe system breakage, that for some reason installation fails and host reboots (I cannot think of a valid sequence), there is another fallback for GSS to extract the logs to a different location by altering a configuration file.

> What is the reason for preferring /tmp here over /var/tmp (at least for
> upstream) which is persisted?

/var/tmp should also be cleared at startup, there should be no real difference between /tmp and /var/tmp except that /tmp is available at boot, while /var/tmp can be  mounted at startup. Both can be cleared at any time, and should be cleared at startup.

--- Additional comment from Stephen Gordon on 2012-12-19 17:15:54 EST ---

(In reply to comment #9)
> (In reply to comment #8)
> > Right, but the proposed resolution to the scenario Simon put forward looks
> > like being to push a very unintuitive procedure off to documentation
> > (otherwise how would the admin know to tweak the setting in
> > /etc/ovirt-host-config.conf)?
> 
> The documentation should state that the logs are collected to the engine
> machine at /var/log/ovirt-engine/host-deploy.
> 
> In cause of severe installation failure, the engine will not be able to
> collect the logs, and the host will not reboot (as there is installation
> failure), in this case there is a fallback to consult logs at /tmp of the
> host machine.
> 
> In case there is a severe system breakage, that for some reason installation
> fails and host reboots (I cannot think of a valid sequence), there is
> another fallback for GSS to extract the logs to a different location by
> altering a configuration file.

OK.

> > What is the reason for preferring /tmp here over /var/tmp (at least for
> > upstream) which is persisted?
> 
> /var/tmp should also be cleared at startup, there should be no real
> difference between /tmp and /var/tmp except that /tmp is available at boot,
> while /var/tmp can be  mounted at startup. Both can be cleared at any time,
> and should be cleared at startup.

The specific difference between them in the FHS is actually that /var/tmp is *not* to be cleared at startup.

--- Additional comment from Alon Bar-Lev on 2012-12-19 17:21:11 EST ---

(In reply to comment #10)
> The specific difference between them in the FHS is actually that /var/tmp is
> *not* to be cleared at startup.

I have no problem to move this to /var/tmp, same-same. will do.

--- Additional comment from Alon Bar-Lev on 2012-12-19 17:25:42 EST ---

(In reply to comment #11)
> (In reply to comment #10)
> > The specific difference between them in the FHS is actually that /var/tmp is
> > *not* to be cleared at startup.
> 
> I have no problem to move this to /var/tmp, same-same. will do.

Question, is it wise to break the default python implementation of tmpdir search[1]?

[1] http://docs.python.org/2/library/tempfile.html#tempfile.gettempdir

--- Additional comment from Stephen Gordon on 2012-12-20 08:37:09 EST ---

(In reply to comment #12)
> (In reply to comment #11)
> > (In reply to comment #10)
> > > The specific difference between them in the FHS is actually that /var/tmp is
> > > *not* to be cleared at startup.
> > 
> > I have no problem to move this to /var/tmp, same-same. will do.
> 
> Question, is it wise to break the default python implementation of tmpdir
> search[1]?
> 
> [1] http://docs.python.org/2/library/tempfile.html#tempfile.gettempdir

Good question, I don't have an answer for that.

--- Additional comment from Alon Bar-Lev on 2012-12-20 08:41:44 EST ---

(In reply to comment #13)
> (In reply to comment #12)
> > (In reply to comment #11)
> > > (In reply to comment #10)
> > > > The specific difference between them in the FHS is actually that /var/tmp is
> > > > *not* to be cleared at startup.
> > > 
> > > I have no problem to move this to /var/tmp, same-same. will do.
> > 
> > Question, is it wise to break the default python implementation of tmpdir
> > search[1]?
> > 
> > [1] http://docs.python.org/2/library/tempfile.html#tempfile.gettempdir
> 
> Good question, I don't have an answer for that.

So I leave this as-is for now. Thanks!

--- Additional comment from Stephen Gordon on 2012-12-20 09:04:48 EST ---

(In reply to comment #14)
> So I leave this as-is for now. Thanks!

Ack, based on previous answers I will clone to docs to ensure that the logging locations (both host and engine) are listed in the appropriate appendix but will leave out the procedure for handling the unlikely/unknown scenario where the host install fails and it reboots without logging to the engine.