Description of problem:

When subvolumes crash, users get messages like "No such file or directory" or "I/O Error" when doing operations that are cluster-wide, i.e., operations that touch the subvolume that has crashed. These include operations like mkdir() and rmdir() which are cluster-wide, as well as reads/writes/creates that hash to the dead subvolume. DHT does the right thing by disallowing operations to the subvolume -- it is effectively putting the subvolume in "read-only" mode to protect data, but it does not return the correct error. As a result, users of the filesystem think that the data is gone (in the case of "No such file or directory", or worse a blanket error that means nothing in the case of EIO). DHT sets the errno to ENOENT, which while makes sense in the context of DHT (No subvolume entry, hence ENOENT), the error it should bubble up to the user is EROFS, since it is putting the system in read-only mode. This diff changes the error messages to EROFS so the users get a more clear message of what is going on. Test Plan: Tested by downing a subvolume and checking error codes. Also ran other prove tests to make sure they pass.

Patch on 3.8 FB branch: https://review.gluster.org/#/c/17952/