Bug 1083476

With regard to the implementation we have two options:

1. enable the mailbox on file master domains always (possible performance hit due the additional IO for monitoring the mailbox)
2. enable the mailbox on file master domains only when there is a block domain in the pool (enabling and disabling the mailbox dynamically is error prone)

In case we are considering 2 we'll have to:

- on connectStoragePool iterate on all the domains to check if at least one is a block domain and eventually activate the mailbox
- when a domain connectivity is restored check if it's a block domain and eventually activate the mailbox (we might have missed it on connectStoragePool)
- when a domain is activated check if it's a block domain and eventually activate the mailbox
- when a domain is deactivated check if it's a block domain and eventually deactivate the mailbox (if no other block domains are left)


At the moment the patch posted upstream is implementing 1. We need to decide if we want (it's worth) to invest time in developing and stabilizing 2.

When there are no active extensions requests the SPM IO required for monitoring the inbox is 1Mb read every 2 seconds (max hosts = 250).

When there are active requests the IO increases (also from the HSM) but obviously this case is not interesting because it means that the mailbox was indeed needed (a block domain in the pool).

(In reply to Federico Simoncelli from comment #2)
> When there are no active extensions requests the SPM IO required for
> monitoring the inbox is 1Mb read every 2 seconds (max hosts = 250).
> 
> When there are active requests the IO increases (also from the HSM) but
> obviously this case is not interesting because it means that the mailbox was
> indeed needed (a block domain in the pool).

Considering the low performance impact tested on a single host, and where we are in the release cycle (approaching RC) we can go ahead with the first approach to always enable the mailbox on file master domains.

This will allow also for QA to test it our on multiple hosts, and provide further indication.

Sean

2 comments:

1 - I would go on #1 above for simplicity (at this point), and try to reduce the IO by simply checking (on file mailboxes) whether the file was changed in the last X seconds (better for the default use case).

2 - 250 hosts is the mailbox limitations.
It may hit us on currently deployed scaled environments on upgrade to 3.4, as this actually activates the mailbox on NFS (IIUC more commonly used on scaled environments)  


E.G. in our currently scale lab we are running 500 fake hosts on NFS pool

(In reply to Barak from comment #4)
> 2 comments:
> 
> 1 - I would go on #1 above for simplicity (at this point), and try to reduce
> the IO by simply checking (on file mailboxes) whether the file was changed
> in the last X seconds (better for the default use case).
> 
> 2 - 250 hosts is the mailbox limitations.
> It may hit us on currently deployed scaled environments on upgrade to 3.4,
> as this actually activates the mailbox on NFS (IIUC more commonly used on
> scaled environments)

I'm do not fully know what are the implications of increasing it,
It can be a simple change in configuration.

Fede ?

> 
> 
> E.G. in our currently scale lab we are running 500 fake hosts on NFS pool

(In reply to Barak from comment #4)
> 2 comments:
> 
> 1 - I would go on #1 above for simplicity (at this point), and try to reduce
> the IO by simply checking (on file mailboxes) whether the file was changed
> in the last X seconds (better for the default use case).

I'm totally on board with this.
IIUC, the impact is adding an additional constant 1MB read every two seconds - nothing to be alarmed about.

If anything, the future optimization should be to completely remove the mailbox when SPM is removed, and have every host serve its own extends.

(In reply to Barak from comment #5)
> (In reply to Barak from comment #4)
> > 2 comments:
> > 
> > 1 - I would go on #1 above for simplicity (at this point), and try to reduce
> > the IO by simply checking (on file mailboxes) whether the file was changed
> > in the last X seconds (better for the default use case).

Good idea to check the file modification timestamp, we could try to evaluate the risk of that change using a different patch on top of #1.

> > 2 - 250 hosts is the mailbox limitations.
> > It may hit us on currently deployed scaled environments on upgrade to 3.4,
> > as this actually activates the mailbox on NFS (IIUC more commonly used on
> > scaled environments)
> 
> I'm do not fully know what are the implications of increasing it,
> It can be a simple change in configuration.

Currently maxHostID is sent from engine to vdsm using the value configured in MaxNumberOfHostsInStoragePool:

fn_db_add_config_value('MaxNumberOfHostsInStoragePool','250','general');

Polling size = MaxNumberOfHostsInStoragePool * 4K = 1Mb (by default)

In the release notes about the mailbox on file domains we should mention that an higher value of MaxNumberOfHostsInStoragePool will trigger larger reads.

verified using av7

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-0504.html