Hide Forgot
When we upgraded from 3.0.4 to 3.1.2, our /var/lib/glusterd was not present on the system. We had re-installed those boxes, and only after installing 3.1.2 did we put the /var/lib/glusterd directory back and start up the volume. However, that resulted in a missing .glusterfs/indices/dirty directory, and all kinds of "0-dict" error messages in the logs. The fix was to set the volume self-heal attribute. This seems a bit problematic that the volume was started without such a required environment in place.
Upgrading with an empty /var/lib/glusterd is something we don't support. Could you explain what's your use case here?
We originally had RHEL 6.6 and upgraded to RHEL 7.2, and had backed up the /var/lib/glusterd. We did not put the /var/lib/glusterd back after the OS upgrade, but only after installing RHGS 3.1.2 from scratch. It would be really helpful if Gluster detected the bad setup and did not run. Loss of data could result, right? Do you think just because it not supported Gluster should still be allowed to execute with a bad configuration?
(In reply to Peter Portante from comment #0) > When we upgraded from 3.0.4 to 3.1.2, our /var/lib/glusterd was not present > on the system. We had re-installed those boxes, and only after installing > 3.1.2 did we put the /var/lib/glusterd directory back and start up the > volume. Did you ensure to restart glusterd to reflect the configuration changes (after putting up content of /var/lib/glusterd) before starting up the volume? > > However, that resulted in a missing .glusterfs/indices/dirty directory, and > all kinds of "0-dict" error messages in the logs. > > The fix was to set the volume self-heal attribute. > > This seems a bit problematic that the volume was started without such a > required environment in place.
(In reply to Atin Mukherjee from comment #4) > (In reply to Peter Portante from comment #0) > > When we upgraded from 3.0.4 to 3.1.2, our /var/lib/glusterd was not present > > on the system. We had re-installed those boxes, and only after installing > > 3.1.2 did we put the /var/lib/glusterd directory back and start up the > > volume. > > Did you ensure to restart glusterd to reflect the configuration changes > (after putting up content of /var/lib/glusterd) before starting up the > volume? I believe so, but don't have a record of this to confirm.
In that case, ideally it should have not been a problem. I will request one of our QE to test this through and then update the BZ. Byreddy - can you please do the needful?
I would think that relying on the customer to do the right thing to prevent data loss is bad for Red Hat. We need to do all we can to make our products simple to use without causing the customer a problem. Having Gluster be sure it does not start without having all its required environment defined correctly would be really helpful. If gluster is not doing that today, wouldn't we want to make sure we do that going forward, regardless of the outcome of the cause of this bug?
Peter, The issue is having an empty /var/lib/glusterd is a very valid case. Imagine when glusterd comes up for the first time it'd not have any configuration stored. So failing on an empty /var/lib/glusterd is not the right way and it's a prerequisite that content o /var/lib/glusterd should not be tampered from the backend until and unless there is some workaround required. In your case, if you have put back the backed up /var/lib/glusterd in the same place post upgrade, things still might fail as volfile regeneration (as part of upgrade) process hasn't happened. We actually saw a similar case because of volfile regenration failed during upgrade path. So if you indeed need to restore the content of /var/lib/glusterd post upgrade here is what you need to do: 1. Upgrade 2. Restore /var/lib/glusterd/ 3. kill glusterd service if its running 4. glusterd --xlator-option upgrade=on -N 5. pkill glusterd 6. start glusterd service Let me know if you have any further questions. IMO, this BZ can be closed. Please do let me know your thought before I do that.
Gluster needs to be sure it does not operate in a mode where it can loose customer data. If you feel in this case gluster operated in way that it could not loose customer data, we can close. However, it does not seem to be the case based on our experience. It is our view point that this is still a problem and needs to be addressed.