Bug 773685

Note that Vdsm does not report whether it is Up or not - it reports its capabilities, and RHEV-M decides if it is worthy of its cluster.

When the second hdd is offline, what's missing from the OS perspective? Which mountpoint is missing? How can vdsm know that a harddisk exists, but offline?

(In reply to comment #2)
> Note that Vdsm does not report whether it is Up or not - it reports its
> capabilities, and RHEV-M decides if it is worthy of its cluster.
> 
> When the second hdd is offline, what's missing from the OS perspective? Which
> mountpoint is missing? How can vdsm know that a harddisk exists, but offline?

Haven't looked too deeply into this, but /data was definitely gone. Depending on the partitioning layout and the amount of local drives, this can be any of the mounts.
I think we need some kind of general health check that would confirm we are able to run in a cluster, and getVdsCaps is missing some params there. In this particular case, the host had to use local storage, but failed to create a local SD because /data wasn't readable (without the Data LV, it's part of the r/o root)
The test for this is easy enough - run mount -a and check for $?

if only /data wasn't available then it's only interesting when user wants to create local storage.  This does not and should not render the host non-operational as the node can be used in a cluster just fine.

When you try to create the storage domain what error do you get?

(In reply to comment #4)
> if only /data wasn't available then it's only interesting when user wants to
> create local storage.  This does not and should not render the host
> non-operational as the node can be used in a cluster just fine.
> 
> When you try to create the storage domain what error do you get?

/data is r/o, vdsm fails on that.

But you're missing the point here, if you want to actually check every resorce for cluster compatibility, then yes, some mountpoints are more relevant to specific use cases than others, but /data is also the place for iso uploading and for host upgrades, unless I'm mistaken. Are those also not critical?

(In reply to comment #5)
> (In reply to comment #4)
> > if only /data wasn't available then it's only interesting when user wants to
> > create local storage.  This does not and should not render the host
> > non-operational as the node can be used in a cluster just fine.
> > 
> > When you try to create the storage domain what error do you get?
> 
> /data is r/o, vdsm fails on that.
> 
> But you're missing the point here, if you want to actually check every resorce
> for cluster compatibility, then yes, some mountpoints are more relevant to
> specific use cases than others, but /data is also the place for iso uploading
> and for host upgrades, unless I'm mistaken. Are those also not critical?

The ISO uploader has nothing to do with vdsm, it's a hack which pushes things into the iso domain using ssh.  vdsm doesn't care about that.
host upgrade on rhev-h means that if we want to upgrade the host it will fail, it doesn't mean that the host cannot currently function as a hypervisor (it can), so no, it is not critical.

note that this is not to say that this shouldn't be checked, but it is definitely no reason to move a host to non-operational.  This is at most a warrning.

Now that I'm clear on the problem and it's implications, I'm changing this to be an rfe and moving upstream as this will not make it into 3.1.

TODO:
1. Add /data to monitored paths in getVdsStats
2. make sure that monitoring includes 'liveness' and not only free space

(In reply to comment #6)
> The ISO uploader has nothing to do with vdsm, it's a hack which pushes things
> into the iso domain using ssh.  vdsm doesn't care about that.
> host upgrade on rhev-h means that if we want to upgrade the host it will fail,
> it doesn't mean that the host cannot currently function as a hypervisor (it
> can), so no, it is not critical.
> 
> note that this is not to say that this shouldn't be checked, but it is
> definitely no reason to move a host to non-operational.  This is at most a
> warrning.

First of all yes, the host is in trouble and we should report that, instead of waiting for something to fail.
Second, this time it was /data that failed to mount, in another config, it might be another mountpoint. We need to assess which mountpoints are critical and start monitoring them, besides reporting issues with the less critical ones.


> 
> Now that I'm clear on the problem and it's implications, I'm changing this to
> be an rfe and moving upstream as this will not make it into 3.1.
> 
> TODO:
> 1. Add /data to monitored paths in getVdsStats
> 2. make sure that monitoring includes 'liveness' and not only free space

I'm not sure this is the right approach here, at least not until we're done assessing the above.

(In reply to comment #7)
> (In reply to comment #6)
> > The ISO uploader has nothing to do with vdsm, it's a hack which pushes things
> > into the iso domain using ssh.  vdsm doesn't care about that.
> > host upgrade on rhev-h means that if we want to upgrade the host it will fail,
> > it doesn't mean that the host cannot currently function as a hypervisor (it
> > can), so no, it is not critical.
> > 
> > note that this is not to say that this shouldn't be checked, but it is
> > definitely no reason to move a host to non-operational.  This is at most a
> > warrning.
> 
> First of all yes, the host is in trouble and we should report that, instead of
> waiting for something to fail.
> Second, this time it was /data that failed to mount, in another config, it
> might be another mountpoint. We need to assess which mountpoints are critical
> and start monitoring them, besides reporting issues with the less critical
> ones.
> 
> 
> > 
> > Now that I'm clear on the problem and it's implications, I'm changing this to
> > be an rfe and moving upstream as this will not make it into 3.1.
> > 
> > TODO:
> > 1. Add /data to monitored paths in getVdsStats
> > 2. make sure that monitoring includes 'liveness' and not only free space
> 
> I'm not sure this is the right approach here, at least not until we're done
> assessing the above.

Assessing whether a mount point is critical is the engine's job when it decided to put a host as non-operational or not.
It is vdsm's job to give it all the data it needs to make a smart decision.
There are cases where it is clear cut, but those are few and not policy driven.

The same host when attached to 2 different clusters could be fine in 1 but non-operational in the other.
I'll give another example to make this clear:
A DC which connects to LUNs over FCP -
 Host A which has no FC HBA's would be non-operational in this DC
Another DC which connects to NFS -
 Same host A could work just fine here.

Same goes to /data

This is exactly why I am saying vdsmd should __report__ additional host metrics. The assessment we need to do is on what exactly needs to be added to reporting, once that additional data is reported, the engine can start looking at it.

So start adding here everything you think should be monitored (in general though, we should just have a set of counters the host can monitor and have the engine define which counters it wants to collect).

Closing old bugs. If this issue is still relevant/important in current version, please re-open the bug.