Bug 886709 - PRD32 - bootstrap: fetch logs to engine
PRD32 - bootstrap: fetch logs to engine
Status: CLOSED ERRATA
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine (Show other bugs)
unspecified
Unspecified Unspecified
unspecified Severity unspecified
: ---
: 3.2.0
Assigned To: Alon Bar-Lev
Pavel Stehlik
infra
: Improvement
Depends On: bootstrap-rewrite
Blocks: 889329
  Show dependency treegraph
 
Reported: 2012-12-12 18:16 EST by Alon Bar-Lev
Modified: 2016-02-10 14:29 EST (History)
13 users (show)

See Also:
Fixed In Version:
Doc Type: Enhancement
Doc Text:
Host logs can now be collected from the /var/log/ovirt-engine/host-deploy directory on the engine, and also from the /tmp/ovirt-host-deploy directory on the host.
Story Points: ---
Clone Of:
: 889329 (view as bug list)
Environment:
Last Closed: 2013-06-10 17:27:24 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Infra
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
dyasny: Triaged+


Attachments (Terms of Use)

  None (edit)
Description Alon Bar-Lev 2012-12-12 18:16:51 EST
At the end of bootstrap, fetch log from host into /var/log/ovirt-engine/host-deploy.

This eases problem determination process as all logs reside on engine side.
Comment 1 Alon Bar-Lev 2012-12-12 18:18:48 EST
In most of the cases logs will be collected.

If severe (protocol violation, communication lost) error is detected, then collection is not available, and log file is available at host machine.
Comment 2 Simon Grinberg 2012-12-13 13:17:05 EST
On on hand it looks like an improvements, since now all the logs are being collected in a central location (except for the host logs collected by the log collector tool)

However the collection instructions become more complex to the end user or GSS.

Before this change is was always: 
1. Go to host:/tmp/xxx.log
Now it's 
1. Go to engine-host:/var/log/ovirt-engine/host-deploy and if it's not there then,
2. Go to host:/tmp/xxx.log

Is that correct?
Comment 3 Alon Bar-Lev 2012-12-13 13:26:08 EST
Unless something terrible wrong logs will be at engine. Much easier.

Also, in fedora and others, /tmp/ is cleared after boot... so you don't have anything.

During deployment you get a message were logs is at host and where they are stored at engine. So you know where to look.

Example of events of new implementation:

---

2012-Dec-13, 20:13 Host test1 installed
2012-Dec-13, 20:13 Installing Host test1. Stage: Termination.
2012-Dec-13, 20:13 Installing Host test1. Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20121213201308-10.35.0.73.log'.
2012-Dec-13, 20:13 Installing Host test1. Reboot scheduled.	
2012-Dec-13, 20:13 Installing Host test1. Stage: Pre-termination.
2012-Dec-13, 20:13 Installing Host test1. Reboot scheduled.
2012-Dec-13, 20:13 Installing Host test1. Stage: Closing up.
2012-Dec-13, 20:13 Installing Host test1. Stage: Transaction commit.
2012-Dec-13, 20:13 Installing Host test1. Enrolling certificate.
2012-Dec-13, 20:13 Installing Host test1. Setting time.
2012-Dec-13, 20:13 Installing Host test1. Stage: Misc configuration.
2012-Dec-13, 20:13 Installing Host test1. Stage: Package installation.
2012-Dec-13, 20:13 Installing Host test1. Stage: Transaction setup.
2012-Dec-13, 20:13 Installing Host test1. Hardware supports virtualization.
2012-Dec-13, 20:13 Installing Host test1. Stage: Setup validation.
2012-Dec-13, 20:13 Installing Host test1. Enfocing host reboot.
2012-Dec-13, 20:13 Installing Host test1. Logs at host located at: '/tmp/ovirt-host-deploy-20121213201256.log'.
2012-Dec-13, 20:13 Installing Host test1. Stage: Environment customization.
2012-Dec-13, 20:13 Installing Host test1. Stage: Programs detection.
2012-Dec-13, 20:12 Installing Host test1. Stage: Installation packages setup.
2012-Dec-13, 20:12 Installing Host test1. Stage: Environment setup.
2012-Dec-13, 20:12 Installing Host test1. Stage: Initializing.
2012-Dec-13, 20:12 Installing Host test1. Connected to host 10.35.0.73 with SSH key fingerprint: 74:b2:98:dd:b4:31:e3:ef:48:71:90:aa:71:d4:f1:db.
Comment 4 Simon Grinberg 2012-12-19 07:59:59 EST
(In reply to comment #3)

> 
> During deployment you get a message were logs is at host and where they are
> stored at engine. So you know where to look.
> 
> 2012-Dec-13, 20:13 Host test1 installed
> 2012-Dec-13, 20:13 Installing Host test1. Stage: Termination.
> 2012-Dec-13, 20:13 Installing Host test1. Retrieving installation logs to:
> '/var/log/ovirt-engine/host-deploy/ovirt-20121213201308-10.35.0.73.log'.

Good

> Also, in fedora and others, /tmp/ is cleared after boot... so you don't have anything.

2012-Dec-13, 20:13 Installing Host test1. Logs at host located at: '/tmp/ovirt-host-deploy-20121213201256.log'. 

Not good, since many times if you can't retrieve the logs then you may have to reboot the machine to retrieve network connection (This is the first thing a remote user (without remote console) will do before actually going to the server room to get console access) 

If /tmp is deleted we may need to consider a different location like /var/log
Comment 5 Alon Bar-Lev 2012-12-19 08:50:19 EST
(In reply to comment #4)
> (In reply to comment #3)

> 2012-Dec-13, 20:13 Installing Host test1. Logs at host located at:
> '/tmp/ovirt-host-deploy-20121213201256.log'. 
> 
> Not good, since many times if you can't retrieve the logs then you may have
> to reboot the machine to retrieve network connection (This is the first
> thing a remote user (without remote console) will do before actually going
> to the server room to get console access) 
> 
> If /tmp is deleted we may need to consider a different location like /var/log

/var/log is managed directory, you cannot put there logs of junk.

managed directory = each file has owner (package).
managed directory = each file is monitored and cleaned up (logrotate and similar).

The deployment process is not installed on the machine, hence should not write to /var/log.

/tmp is the right place, as done in the past.

sysadmin can override this location using /etc/ovirt-host-config.conf, if indeed there is a problem during deployment, host reboot but engine cannot fetch log.
Comment 6 Stephen Gordon 2012-12-19 16:57:16 EST
(In reply to comment #4)
> If /tmp is deleted we may need to consider a different location like /var/log

Note that in Fedora 18 after much back and forth the default will be for /tmp to be a tmpfs filesystem. That means it is wiped on reboot. I would expect this to impact upstream ovirt and potentially future versions of RHEV (if this change ever makes it into RHEL).
Comment 7 Alon Bar-Lev 2012-12-19 16:59:48 EST
(In reply to comment #6)
> (In reply to comment #4)
> > If /tmp is deleted we may need to consider a different location like /var/log
> 
> Note that in Fedora 18 after much back and forth the default will be for
> /tmp to be a tmpfs filesystem. That means it is wiped on reboot. I would
> expect this to impact upstream ovirt and potentially future versions of RHEV
> (if this change ever makes it into RHEL).

Known issue, refer to previous comments.

I am not sure it is tmpfs, but sure it is cleared on startup, many distributions already took this approach which is correct.

Thanks!
Comment 8 Stephen Gordon 2012-12-19 17:06:33 EST
Right, but the proposed resolution to the scenario Simon put forward looks like being to push a very unintuitive procedure off to documentation (otherwise how would the admin know to tweak the setting in /etc/ovirt-host-config.conf)?

What is the reason for preferring /tmp here over /var/tmp (at least for upstream) which is persisted?
Comment 9 Alon Bar-Lev 2012-12-19 17:12:19 EST
(In reply to comment #8)
> Right, but the proposed resolution to the scenario Simon put forward looks
> like being to push a very unintuitive procedure off to documentation
> (otherwise how would the admin know to tweak the setting in
> /etc/ovirt-host-config.conf)?

The documentation should state that the logs are collected to the engine machine at /var/log/ovirt-engine/host-deploy.

In cause of severe installation failure, the engine will not be able to collect the logs, and the host will not reboot (as there is installation failure), in this case there is a fallback to consult logs at /tmp of the host machine.

In case there is a severe system breakage, that for some reason installation fails and host reboots (I cannot think of a valid sequence), there is another fallback for GSS to extract the logs to a different location by altering a configuration file.

> What is the reason for preferring /tmp here over /var/tmp (at least for
> upstream) which is persisted?

/var/tmp should also be cleared at startup, there should be no real difference between /tmp and /var/tmp except that /tmp is available at boot, while /var/tmp can be  mounted at startup. Both can be cleared at any time, and should be cleared at startup.
Comment 10 Stephen Gordon 2012-12-19 17:15:54 EST
(In reply to comment #9)
> (In reply to comment #8)
> > Right, but the proposed resolution to the scenario Simon put forward looks
> > like being to push a very unintuitive procedure off to documentation
> > (otherwise how would the admin know to tweak the setting in
> > /etc/ovirt-host-config.conf)?
> 
> The documentation should state that the logs are collected to the engine
> machine at /var/log/ovirt-engine/host-deploy.
> 
> In cause of severe installation failure, the engine will not be able to
> collect the logs, and the host will not reboot (as there is installation
> failure), in this case there is a fallback to consult logs at /tmp of the
> host machine.
> 
> In case there is a severe system breakage, that for some reason installation
> fails and host reboots (I cannot think of a valid sequence), there is
> another fallback for GSS to extract the logs to a different location by
> altering a configuration file.

OK.

> > What is the reason for preferring /tmp here over /var/tmp (at least for
> > upstream) which is persisted?
> 
> /var/tmp should also be cleared at startup, there should be no real
> difference between /tmp and /var/tmp except that /tmp is available at boot,
> while /var/tmp can be  mounted at startup. Both can be cleared at any time,
> and should be cleared at startup.

The specific difference between them in the FHS is actually that /var/tmp is *not* to be cleared at startup.
Comment 11 Alon Bar-Lev 2012-12-19 17:21:11 EST
(In reply to comment #10)
> The specific difference between them in the FHS is actually that /var/tmp is
> *not* to be cleared at startup.

I have no problem to move this to /var/tmp, same-same. will do.
Comment 12 Alon Bar-Lev 2012-12-19 17:25:42 EST
(In reply to comment #11)
> (In reply to comment #10)
> > The specific difference between them in the FHS is actually that /var/tmp is
> > *not* to be cleared at startup.
> 
> I have no problem to move this to /var/tmp, same-same. will do.

Question, is it wise to break the default python implementation of tmpdir search[1]?

[1] http://docs.python.org/2/library/tempfile.html#tempfile.gettempdir
Comment 13 Stephen Gordon 2012-12-20 08:37:09 EST
(In reply to comment #12)
> (In reply to comment #11)
> > (In reply to comment #10)
> > > The specific difference between them in the FHS is actually that /var/tmp is
> > > *not* to be cleared at startup.
> > 
> > I have no problem to move this to /var/tmp, same-same. will do.
> 
> Question, is it wise to break the default python implementation of tmpdir
> search[1]?
> 
> [1] http://docs.python.org/2/library/tempfile.html#tempfile.gettempdir

Good question, I don't have an answer for that.
Comment 14 Alon Bar-Lev 2012-12-20 08:41:44 EST
(In reply to comment #13)
> (In reply to comment #12)
> > (In reply to comment #11)
> > > (In reply to comment #10)
> > > > The specific difference between them in the FHS is actually that /var/tmp is
> > > > *not* to be cleared at startup.
> > > 
> > > I have no problem to move this to /var/tmp, same-same. will do.
> > 
> > Question, is it wise to break the default python implementation of tmpdir
> > search[1]?
> > 
> > [1] http://docs.python.org/2/library/tempfile.html#tempfile.gettempdir
> 
> Good question, I don't have an answer for that.

So I leave this as-is for now. Thanks!
Comment 15 Stephen Gordon 2012-12-20 09:04:48 EST
(In reply to comment #14)
> So I leave this as-is for now. Thanks!

Ack, based on previous answers I will clone to docs to ensure that the logging locations (both host and engine) are listed in the appropriate appendix but will leave out the procedure for handling the unlikely/unknown scenario where the host install fails and it reboots without logging to the engine.
Comment 16 Oded Ramraz 2013-01-28 08:22:18 EST
Verified sf4
Comment 17 Tim Hildred 2013-02-19 02:14:03 EST
(In reply to comment #2)
> Before this change is was always: 
> 1. Go to host:/tmp/xxx.log
> Now it's 
> 1. Go to engine-host:/var/log/ovirt-engine/host-deploy and if it's not there
> then,
> 2. Go to host:/tmp/xxx.log
> 
> Is that correct?

Hey Alon,

I'm trying to resolve BZ#889333, and update "Red Hat Enterprise Virtualization Host Log Files"

https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Virtualization/3.1/html-single/Installation_Guide/index.html#Red_Hat_Enterprise_Virtualization_Host_Log_Files1

Host side installation logs go to /tmp/xxx.log. What is the value of "xxx"?

Thanks for your help.
Comment 18 Alon Bar-Lev 2013-02-19 16:22:49 EST
(In reply to comment #17)
> (In reply to comment #2)
> > Before this change is was always: 
> > 1. Go to host:/tmp/xxx.log
> > Now it's 
> > 1. Go to engine-host:/var/log/ovirt-engine/host-deploy and if it's not there
> > then,
> > 2. Go to host:/tmp/xxx.log
> > 
> > Is that correct?
> 
> Hey Alon,
> 
> I'm trying to resolve BZ#889333, and update "Red Hat Enterprise
> Virtualization Host Log Files"
> 
> https://access.redhat.com/knowledge/docs/en-US/
> Red_Hat_Enterprise_Virtualization/3.1/html-single/Installation_Guide/index.
> html#Red_Hat_Enterprise_Virtualization_Host_Log_Files1
> 
> Host side installation logs go to /tmp/xxx.log. What is the value of "xxx"?
> 
> Thanks for your help.

At host side it is on /tmp/ovirt-host-deploy-@DATE@.log, but once deployment is finished it is copied to engine side to /var/log/ovirt-engine/host-deploy/ovirt-@DATE@-@HOST@-@CORRELATION_ID@.log and removed from host.

Both logs' name is printed at event log during deployment.
Comment 19 Tim Hildred 2013-02-19 20:55:53 EST
Updated:

Red Hat Enterprise Virtualization Manager Log Files [7862, rev:374913]
Red Hat Enterprise Virtualization Host Log Files [9421, rev: 375129]
Comment 20 Cheryn Tan 2013-04-03 02:53:37 EDT
This bug is currently attached to errata RHEA-2013:14491. If this change is not to be documented in the text for this errata please either remove it from the errata, set the requires_doc_text flag to minus (-), or leave a "Doc Text" value of "--no tech note required" if you do not have permission to alter the flag.

Otherwise to aid in the development of relevant and accurate release documentation, please fill out the "Doc Text" field above with these four (4) pieces of information:

* Cause: What actions or circumstances cause this bug to present.

* Consequence: What happens when the bug presents.

* Fix: What was done to fix the bug.

* Result: What now happens when the actions or circumstances above occur. (NB: this is not the same as 'the bug doesn't present anymore')

Once filled out, please set the "Doc Type" field to the appropriate value for the type of change made and submit your edits to the bug.

For further details on the Cause, Consequence, Fix, Result format please refer to:

https://bugzilla.redhat.com/page.cgi?id=fields.html#cf_release_notes

Thanks in advance.
Comment 21 errata-xmlrpc 2013-06-10 17:27:24 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-0888.html

Note You need to log in before you can comment on or make changes to this bug.