Fedora Account System
Red Hat Associate
Red Hat Customer
Description of problem: Script bin/setup_container_executor.sh is launched automatically when enabling new gluster volume for hadoop (from enable_vol.sh). The problem is, that it rewrite existing and configured file /etc/hadoop/conf/container-executor.cfg. User have to stop all Mapreduce/YARN services, reconfigure the container-executor.cfg and start the services again. And if the user decide to add new gluster volume to hadoop after some time of using it (he will have container-executor.cfg properly configured), this behaviour of setup_container_executor.sh will break the cluster and it is not immediately clear where is the problem and why are all mapreduce job failing. Version-Release number of selected component (if applicable): # rpm -q rhs-hadoop-install rhs-hadoop-install-2_11-1.el6rhs.noarch How reproducible: 100% Steps to Reproduce: 1. Configure /etc/hadoop/conf/container-executor.cfg. 2. Enable new gluster volume for hadoop (via enable_vol.sh) 3. Check the configuration file /etc/hadoop/conf/container-executor.cfg. Actual results: Configuration file /etc/hadoop/conf/container-executor.cfg is rewritten by (bad) default configuration. Expected results: Script enable_vol.sh will not rewrite configuration file /etc/hadoop/conf/container-executor.cfg. Additional info: The default configuration for container-executor.cfg (from setup_container_executor.sh) is also bad, because it set min.user.id=1000, when user IDs for ordinary users on RHEL 6 starts with 500.
Use md5sum to decide if the existing container-executor.cfg file has been previously modified vs is the original version of the file. If original then we overwrite it in bin/setup_container_executor.sh. Fixed in version 2.20
If configuration file /etc/hadoop/conf/container-executor.cfg is changed, setup_container_executor.sh doesn't overwrite it, but it also doesn't perform the permission correction for the config file and /usr/lib/hadoop-yarn/bin/container-executor file, which was the initial purpose of this script. I think (and propose), that this permissions correction should/could be done always - it shouldn't break anything because the required permissions are always the same and it can fix some problems with bad permissions on container-executor files. If you think, that current behaviour is correct, fell free to move this BZ back to ON_QA - but then it should be documented how to properly set permissions on affected files related to container executor in case of problems. >> ASSIGNED
Related version for previous comment is: rhs-hadoop-install-2_28-1.el6rhs.noarch
Daniel, I understand your comment #3. But consider case where the customer doesn't want the owner of the container-executor.cfg to be root, or requires different but usable perms on this file, or has a different group for this file? If any of the above scenarios are possible then we should not execute the rest of the bin/setup_container_executor.sh script. It is easy to make the changes you suggest in #3 but is it the right thing to do?
Jeff, your point make sense and I haven't really deep knowledge, but it seems like container-executor.cfg have to be owned by root and must not be world or group writable. Two errors from yarn-yarn-nodemanager-host....redhat.com.log when I've changed the default ownership/permissions on the config file: org.apache.hadoop.util.Shell$ExitCodeException: File /etc/hadoop/conf.empty/container-executor.cfg must not be world or group writable, but is 666 org.apache.hadoop.util.Shell$ExitCodeException: File /etc/hadoop/conf.empty/container-executor.cfg must be owned by root, but is owned by 501 And I think that we don't support different group than "hadoop", do we? So I'm not sure if there is any valid case, when it might be required to set different permissions than the default one?
Hi Daniel, The consensus from the "big data" team is to not change the container-executor.cfg file if it has been modified in anyway by the customer. enable_vol.sh logs that the file has been modified and is not being changed.
Wait, are we sure about this? Consider the following scenario: Admin edits /etc/hadoop/conf/container-executor.cfg and changes min.user.id (without changing the access rights on the file), then he runs the installer scripts. The result would be broken cluster, because the access rights on the cfg file hasn't been tweaked for rhs-hadoop. So I have two questions: * Do I understand this issue right? * Why we would not support the use case I mentioned? The current behaviour which allows such breakage looks like a bug to me.
I guess Daniel answered this question in comment #6: what happens if the user changes the owner to be non-root? The answer appears to be this error "/container-executor.cfg must be owned by root, but is owned by 501". This error would appear to indicate that the owner of the container-executor.cfg file must always be root. So, if the user changes the owner to non-root and then runs enable_vol which changes the owner back to root, this seems to be ok (and even necessary). Note: the perms on the container-executor.cgf file are not modified by enable_vol. But we are dealing with 2 files here: /etc/hadoop/conf/container-executor.cfg and /usr/lib/hadoop-yarn/bin/container-executor. The later also has its owner changed to root:hadoop, and, in addition, has its perms changed to 6050. Has any testing been done on user changes to /usr/lib/hadoop-yarn/bin/container-executor ? enable_vol is NOT sensitive to changes made to /usr/lib/hadoop-yarn/bi/container-executor. And, /usr/lib/hadoop-yarn/bin/container-executor is not chown'd or chmod'ed if the container-executor.cfg file is modified, even if the former has not been modified. The script seems inconsistent...
First of all - this cannot be CLOSED as NOTABUG, because originally it was about always rewriting container-executor.cgf by setup_container_executor.sh script and this behaviour was changed - so it is a bug which have to be properly verified (if not more changed). When I think about it deeply and check it slightly, it seems like there is no other possible working ownership/permissions configuration different that the default one for container-executor and container-executor.cfg files, so why we didn't set/correct the ownership and permissions always and avoid strange failures and problems? Some errors are quite clear - like the two mentioned above, but they are only in log file yarn-yarn-nodemanager-<hostname>.com.log, so it is not clear what is wrong just from the output of failing job. Another errors related to bad ownership/permissions are directly in the log of failing job, but it is not clear which permissions have to be set (for example permissions 6050 for container-executor are slightly unusual - and they are not documented anywhere in our documentation). If you are still decided to leave it as it is now - (do not change anything if container-executor.cfg was changed), please move this bug to MODIFIED and we will verify it.
The only variable could be the group name 'hadoop' which could change if the hadoop daemon user changed. However, changing the hadoop daemon user/group is far outside the scope of support for this script. It would be safe forcing owner:group/permissions for these two files regardless of the contents and only modify the container_executor.ini if its the original.
Jeff, I think your comment 13 was designated for bug 1142212, so the issue of this bug is still valid (setting owner,group and permissions to container executor related files).
Daniel, thanks for catching this, you are correct in comment 14. Per Brad's comment 11, I will make the changes suggested in this BZ.
Tested and VERIFIED on Red Hat Storage Server 3.0 Update 4 with rhs-hadoop-install-2_44-1.el6rhs.noarch. Script bin/setup_container_executor.sh properly check if /etc/hadoop/conf/container-executor.cfg was changed or not and replace it only if not. Also ownership and permissions of /etc/hadoop/conf/container-executor.cfg and /usr/lib/hadoop-yarn/bin/container-executor are correctly fixed. >> VERIFIED
Jeff, Could you please provide the doc text in the Doc Text field for this bug?
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHEA-2015-0761.html