Bug 1103973

Darshan, can you check what glusterVolumesList vdsm command returns. If this returns the brick status correctly, it may be an engine issue.

GlusterVolumeStatus command is returning the brick status correctly.

Is there any progress/workaround? I see exactly the same problem with oVirt 3.4.4 + gluster 3.5.2:

engine: ovirt-engine-3.4.4-1.el6.noarch
hosts: vdsm-4.14.17-0.el6.x86_64, vdsm-gluster-4.14.17-0.el6.noarch, glusterfs-server-3.5.2-1.el6.x86_64

How is the hostname in glusterVolumeStatus gathered?
I have multiple nics in host machines (gluster listening on 0.0.0.0) with:
vm1.lab:
eth1: 192.168.254.1/30
ovirtmgmt (bridge on eth2): 192.168.254.129/27, 192.168.254.161/27
$ grep vm1 /etc/hosts
192.168.254.1    vm1.lab.host
192.168.254.129  vm1.lab.gluster

vm2.lab:
eth1: 192.168.254.5/30
ovirtmgmt (bridge on eth2): 192.168.254.130/27, 192.168.254.162/27
$ grep vm2 /etc/hosts
192.168.254.5    vm2.lab.host
192.168.254.130  vm2.lab.gluster

I'm not aware of any hosts IP-Name change related to this. However I also modified the /etc/hosts during oVirt upgrade.

And glusterVolumeStatus output:

$ vdsClient -s localhost glusterVolumeStatus volumeName=storage
{'status': {'code': 0, 'message': 'Done'},
 'volumeStatus': {'bricks': [{'brick': 'vm1.lab.gluster:/gluster/vms/storage',
                              'hostuuid': '0c779c52-a097-4101-9c85-c9636499ce82',
                              'pid': '1894',
                              'port': '50153',
                              'status': 'ONLINE'},
                             {'brick': 'vm2.lab.gluster:/gluster/vms/storage',
                              'hostuuid': '21a5bd1e-78e3-4824-b299-f7a7c72b7d7a',
                              'pid': '1758',
                              'port': '50159',
                              'status': 'ONLINE'}],
                  'name': 'storage',
                  'nfs': [{'hostname': '192.168.254.5',
                           'hostuuid': '21a5bd1e-78e3-4824-b299-f7a7c72b7d7a',
                           'pid': '24817',
                           'port': '2049',
                           'status': 'ONLINE'},
                          {'hostname': '192.168.254.129',
                           'hostuuid': '0c779c52-a097-4101-9c85-c9636499ce82',
                           'pid': '16277',
                           'port': '2049',
                           'status': 'ONLINE'}],
                  'shd': [{'hostname': '192.168.254.5',
                           'hostuuid': '21a5bd1e-78e3-4824-b299-f7a7c72b7d7a',
                           'pid': '24824',
                           'status': 'ONLINE'},
                          {'hostname': '192.168.254.129',
                           'hostuuid': '0c779c52-a097-4101-9c85-c9636499ce82',
                           'pid': '16286',
                           'status': 'ONLINE'}]}}

The previous output is from vm2.lab.gluster host.

Now here is the output from vm1.lab.gluster (doesn't match =
Local IP is wrong, remote IP is ok)/

$ vdsClient -s localhost glusterVolumeStatus volumeName=storage
{'status': {'code': 0, 'message': 'Done'},
 'volumeStatus': {'bricks': [{'brick': 'vm1.lab.gluster:/gluster/vms/storage',
                              'hostuuid': '0c779c52-a097-4101-9c85-c9636499ce82',
                              'pid': '1894',
                              'port': '50153',
                              'status': 'ONLINE'},
                             {'brick': 'vm2.lab.gluster:/gluster/vms/storage',
                              'hostuuid': '21a5bd1e-78e3-4824-b299-f7a7c72b7d7a',
                              'pid': '1758',
                              'port': '50159',
                              'status': 'ONLINE'}],
                  'name': 'storage',
                  'nfs': [{'hostname': '192.168.254.1',
                           'hostuuid': '0c779c52-a097-4101-9c85-c9636499ce82',
                           'pid': '16277',
                           'port': '2049',
                           'status': 'ONLINE'},
                          {'hostname': '192.168.254.130',
                           'hostuuid': '21a5bd1e-78e3-4824-b299-f7a7c72b7d7a',
                           'pid': '24817',
                           'port': '2049',
                           'status': 'ONLINE'}],
                  'shd': [{'hostname': '192.168.254.1',
                           'hostuuid': '0c779c52-a097-4101-9c85-c9636499ce82',
                           'pid': '16286',
                           'status': 'ONLINE'},
                          {'hostname': '192.168.254.130',
                           'hostuuid': '21a5bd1e-78e3-4824-b299-f7a7c72b7d7a',
                           'pid': '24824',
                           'status': 'ONLINE'}]}}

The host names returned from gluster volume status is mapped to engine's host using the hostuuid field.

In your engine, the hosts that you have:
vm1.lab.gluster
vm2.lab.gluster, could you tell me the host uuid for these.

You could do this, by running the query:
psql engine postgres -c "select vds_name, gluster_server_uuid  from vds_static, gluster_server where vds_id= server_id;"

Also, could you attach the engine.log to the bug?

Created attachment 942592 [details]
log from ovirt hosted engine

Created attachment 942593 [details]
log from vm1 ovirt host

Created attachment 942594 [details]
log from vm2 ovirt host

I attached requested log (and also vdsm logs from nodes). I'm running two node oVirt with hosted engine. The hosts are also gluster nodes. All gluster volumes are replica with two bricks.

They're whole day logs. I don't remember exactly when I started the upgrade. But definitely I upgraded in order:
1) ovirt-engine
2) vm1.lab
3) vm2.lab

The gluster worked perfectly all the time. What I describe below is just the status in oVirt admin console.

The status of gluster volumes in oVirt was ok in the morning. After that (and I don't remember if after 1st or 2nd step - but I think after 2nd step) the gluster bricks on vm1 went down (red triangle). Than I upgraded also vm2.lab. All gluster bricks on vm1 were red and on vm2 there question marks (no red no green triangle just black question mark). As the last thing I tried to stop & start isos brick from admin console and it went into green status.

In the meanwhile I tried restart whole environment (engine, hosts) without impact to the running VMs. Also I tried stop & start isos and it went into green status but after some another step it went into vm1 red and vm2 question mark state.

I'm trying to run pgsql query but I was never logged to the ovirt internal pgsql instance (so I don't know credentials - or where to get them - and I don't have identd installed). So it'll take me some time. Maybe I'll change ident to trust auth :)

# sudo -u postgres psql engine postgres -c "select vds_name, gluster_server_uuid  from vds_static, gluster_server where vds_id= server_id;"
   vds_name      |         gluster_server_uuid          
-----------------+--------------------------------------
 vm1.lab.gluster | 0c779c52-a097-4101-9c85-c9636499ce82
 vm2.lab.gluster | 21a5bd1e-78e3-4824-b299-f7a7c72b7d7a
(2 rows)

I see how to reproduce the question mark state:
1) put the node under maintenance (not sure if needed)
2) stop vdsmd service on that node
3) Refresh capabilities from web admin console for that node while vdsmd is down

I see hot to reproduce the down (red triangle) state:
1) put the node under maintenance (not sure if needed)
2) stop glusterd service on that node
3) Refresh capabilities from web admin console for that node while glusterd is down

Hi!

Thanks for the detailed analysis on the bug.

This is designed as per bug - https://bugzilla.redhat.com/show_bug.cgi?id=1021441#c4

1) In the first case - when vdsmd is down, there is no communication possible between the engine and node. This could be due to many reasons - host powered down or vdsmd service not running. So the brick status is temporarily moved to Unknown (?) - the brick will be moved back to UP state, during the next refresh cycle when gluster volume status returns the brick as online.

2) In the second case - when glusterd is down, the bricks are marked Down (red) since gluster volume status will no longer list these bricks.


If the UNKNOWN (?) state is misleading, we could change our refresh logic - to always compare results from gluster volume status output. If the brick is not listed in the output, then mark this as DOWN.

Please let me know if the UNKNOWN state was the issue. (As the second case seems to be expected behaviour)

States are ok and I understand it. I have no problem with it.

There is no problem that it went into that state. The problem is that it never returns from that state.
It's in that status one day (with vdsmd and glusterd up). Also Refresh capabilities doesn't help.

The Gluster status refresh is weird. Right now I'm after series of test in situation that vm1 is down (host is up but all services includind wdmd, sanlock, gluster, vdsm is down) - for longer time than 2 hours but I see in ovirt console that brick on vm1 is up and brick on vm2 is down (opposite to the reality).

I see how to reproduce the question mark state:
1) put the node under maintenance (not sure if needed)
2) stop vdsmd service on that node
3) Refresh capabilities from web admin console for that node while vdsmd is down
4) start vdsmd service on that node
5) brick remain in unknown state even Refresh capabilities doesn't help (the only possible transition is to down state)

I see hot to reproduce the down (red triangle) state:
1) put the node under maintenance (not sure if needed)
2) stop glusterd service on that node
3) Refresh capabilities from web admin console for that node while glusterd is down
4) start glusterd service on that node
5) brick remain in down state even Refresh capabilities doesn't help

The problem arrived yesterday and I still see the strange state.
The only way from this is to stop & start gluster volume from ovirt console.

(In reply to Sahina Bose from comment #13)
> Hi!
> 
> Thanks for the detailed analysis on the bug.
> 
> This is designed as per bug -
> https://bugzilla.redhat.com/show_bug.cgi?id=1021441#c4
> 
> 1) In the first case - when vdsmd is down, there is no communication
> possible between the engine and node. This could be due to many reasons -
> host powered down or vdsmd service not running. So the brick status is
> temporarily moved to Unknown (?) - the brick will be moved back to UP state,
> during the next refresh cycle when gluster volume status returns the brick
> as online.

per my original bug report, agreed this is the expected behaviour but this does not happen in my cluster, is this resolved in 3.5? Until this additional report came in I had not seen any action on this bug, or even a confirmation that there was a problem, or a request for information.

> 
> 2) In the second case - when glusterd is down, the bricks are marked Down
> (red) since gluster volume status will no longer list these bricks.
> 
> 
> If the UNKNOWN (?) state is misleading, we could change our refresh logic -
> to always compare results from gluster volume status output. If the brick is
> not listed in the output, then mark this as DOWN.
> 
> Please let me know if the UNKNOWN state was the issue. (As the second case
> seems to be expected behaviour)

(In reply to Alastair Neil from comment #16)
> (In reply to Sahina Bose from comment #13)
> > Hi!
> > 
> > Thanks for the detailed analysis on the bug.
> > 
> > This is designed as per bug -
> > https://bugzilla.redhat.com/show_bug.cgi?id=1021441#c4
> > 
> > 1) In the first case - when vdsmd is down, there is no communication
> > possible between the engine and node. This could be due to many reasons -
> > host powered down or vdsmd service not running. So the brick status is
> > temporarily moved to Unknown (?) - the brick will be moved back to UP state,
> > during the next refresh cycle when gluster volume status returns the brick
> > as online.
> 
> per my original bug report, agreed this is the expected behaviour but this
> does not happen in my cluster, is this resolved in 3.5? Until this
> additional report came in I had not seen any action on this bug, or even a
> confirmation that there was a problem, or a request for information.


This fix was introduced in http://gerrit.ovirt.org/#/c/21444/ and should be available since ovirt-3.4.0

We had tried to reproduce the issue, but were unable to. We may be missing a scenario here.

If this happens again, could you provide the engine log and vdsm log?

> 
> > 
> > 2) In the second case - when glusterd is down, the bricks are marked Down
> > (red) since gluster volume status will no longer list these bricks.
> > 
> > 
> > If the UNKNOWN (?) state is misleading, we could change our refresh logic -
> > to always compare results from gluster volume status output. If the brick is
> > not listed in the output, then mark this as DOWN.
> > 
> > Please let me know if the UNKNOWN state was the issue. (As the second case
> > seems to be expected behaviour)

(In reply to Ludek Finstrle from comment #15)
> I see how to reproduce the question mark state:
> 1) put the node under maintenance (not sure if needed)
> 2) stop vdsmd service on that node
> 3) Refresh capabilities from web admin console for that node while vdsmd is
> down
> 4) start vdsmd service on that node
> 5) brick remain in unknown state even Refresh capabilities doesn't help (the
> only possible transition is to down state)
> 
> I see hot to reproduce the down (red triangle) state:
> 1) put the node under maintenance (not sure if needed)
> 2) stop glusterd service on that node
> 3) Refresh capabilities from web admin console for that node while glusterd
> is down
> 4) start glusterd service on that node
> 5) brick remain in down state even Refresh capabilities doesn't help
> 
> The problem arrived yesterday and I still see the strange state.
> The only way from this is to stop & start gluster volume from ovirt console.

The refresh capabilities only executes the getVdsCaps on the node. It does not execute the gluster commands - hence would not result in changing brick state to green. I did not notice any error in your logs regarding the gluster commands. Let me dig into this further now that there are reproducible steps.

As the logs of engine and vdsm were not from the same time, there was not much I could infer from the logs. No exceptions related to volumestatus found in either.

One possibility is that brick status does not get updated due to host names differing - the one that gluster returns and the one engine is aware of. The referenced patch addresses this issue.

Ops, I'm sorry I didn't notice that log rotation proceeded while I downloaded logs.
Are you interested in vdsm logs from 9/29/2014?

Will this patch be included in 3.5.0 release or 3.5.1?

(In reply to Ludek Finstrle from comment #20)
> Ops, I'm sorry I didn't notice that log rotation proceeded while I
> downloaded logs.
> Are you interested in vdsm logs from 9/29/2014?

If you do have it, yes. And this is the time during which the brick status was in red state, correct?

> 
> Will this patch be included in 3.5.0 release or 3.5.1?

It should be in 3.5.0

Created attachment 943369 [details]
vdsm log from vm1

Created attachment 943370 [details]
vdsm log from vm1

Created attachment 943371 [details]
vdsm log from vm1

Created attachment 943372 [details]
vdsm log from vm1

Created attachment 943373 [details]
vdsm log from vm2

Created attachment 943374 [details]
vdsm log from vm2

Created attachment 943375 [details]
vdsm log from vm2

Created attachment 943376 [details]
vdsm log from vm2

Created attachment 943377 [details]
vdsm log from vm2

  I upploaded vdsm logs from vm1 and vm2 hopefully from the right time.
Previously attached vdsm logs (from 9/30) are from time when the status of gluster volumes is displayed wrong (red for all except isos - I stopped & started isos).

oVirt 3.5 has been released and should include the fix for this issue.

I can confirm that this issue is resolved for me with verion 3.5GA