Red Hat Bugzilla – Bug 1408172
[networking_public_267]'Permission denied' for check script and notify script for ipfailover pod
Last modified: 2017-07-24 10:11 EDT
Description of problem: Create ipfailover pod with check and notify script. there is an error "Can't open /root/notify.sh (errno 2 No such file or directory)" ssh into pod and also did not find the file # oc rsh ipf-1-j54id ls /root anaconda-ks.cfg Version-Release number of selected component (if applicable): openshift v1.5.0-alpha.0+be217b0-499 kubernetes v1.4.0+776c994 etcd 3.1.0-rc.0 I re-build the keeplived image with updated file. https://github.com/openshift/origin/pull/11644/files#diff-f67c9d89133c83f4d1e56503d11501b3R44 - HA_VRRP_ID_OFFSET="${OPENSHIFT_HA_VRRP_ID_OFFSET:-"0""}" + HA_VRRP_ID_OFFSET="${OPENSHIFT_HA_VRRP_ID_OFFSET:-"0"}" How reproducible: always Steps to Reproduce: 1. touch 2 file '/root/check.sh' and '/root/notify.sh' 2. Create ipfailover pod with oadm ipfailover ipf --create --virtual-ips='172.18.3.100' --replicas=1 -w 80 --check-interval=4 --check-script='/root/check.sh' --latest-images --notify-script='/root/notify.sh' 3. Check the pod logs 4. check the file in the pod container Actual results: step 3; oc logs $pod VRRP_Instance(ipf_VIP_1) Entering MASTER STATE VRRP_Instance(ipf_VIP_1) setting protocol VIPs. VRRP_Instance(ipf_VIP_1) Sending gratuitous ARPs on eth0 for 172.18.3.100 VRRP_Group(group_ipf) Syncing instances to MASTER state Opening script file /root/notify.sh Can't open /root/notify.sh (errno 2 No such file or directory) Netlink reflector reports IP 172.18.3.100 added step 4; # oc rsh ipf-1-j54id ls /root anaconda-ks.cfg Expected results: Check script and notify script can work for ipfailover pod. Additional info:
There is a problem in setting up the test. Step 1: touch 2 file '/root/check.sh' and '/root/notify.sh' creates the 2 files on the host not in the pod. The files must be in the pod for this to work. This can be done with a configmap, or mounted through a PV/PVC, or by modifying the images/ipfailover/keepalived/Dockerfile (rebuild openshift/origin, ipfailover/keepalived, push to default registry, edit dc to reference the new image). In the case of the configmap or mount, the dc needs to be edited to mount the file/configmap. In the case of the Docker file change, the image needs to be pushed to the default registry in the cluster and the image in the dc needs to point to it. See openshift/openshift-docs/pr/3355
hi ,Phil I re-tested this using the steps in #PR 3355 those two script can be mounted into pod. but 'Permission denied' for those two file steps: 1) create two files /root/check.sh and /root/notify.sh 2) add permission " chmod +x /root/*.sh" 3) Create ipfailover pod oadm ipfailover ipf --create --virtual-ips='172.18.3.100' --replicas=1 -w 80 --check-interval=4 --check-script='/root/check.sh' --latest-images --notify-script='/root/notify.sh' 4) oc create configmap mycustomcheck --from-file=/root/ 5) oc volume dc/ipf --add --overwrite --name=config-volume --mount-path=/etc/keepalive/ --source='{"configMap": { "name": "mycustomcheck"}}' 6) oc rsh ipf_xxx sh-4.2#ls /etc/keepalive check.sh notify.sh 7) sh-4.2# ./check.sh sh: ./check.sh: Permission denied when I updated the config to: if [[ -f /root/check.sh ]]; then - script "/root/check.sh" + script "sh /root/check.sh" fi it can work
zzhao@redhat.com Thanks, I will make the change. phil
Commit pushed to master at https://github.com/openshift/origin https://github.com/openshift/origin/commit/39fc8f0f8ef19a96fd9701d4ce7394accb91c129 Check and notify scripts are not executable The ipfailover check and notify scripts must be present and executable in the pod. The recommended configmap (doc PR 3355) places the files in the pod but does not set the execute bit. This change sets the execute bit in the scripts. Bug: 1408172 https://bugzilla.redhat.com/show_bug.cgi?id=1408172 Signed-off-by: Phil Cameron <pcameron@redhat.com>
hi, Phil I did testing on images " openshift/origin-keepalived-ipfailover latest 6da1e099dca9 " the issue still can be reproduced when the check script failed and rollback reproduce steps: 1) create two files /root/check.sh and /root/notify.sh cat check.sh </dev/tcp/172.30.157.37/80 ##router service ip/port 2) Create ipfailover pod oadm ipfailover ipf --create --virtual-ips='172.18.3.100' --replicas=1 -w 80 --check-interval=4 --check-script='/root/check.sh' --latest-images --notify-script='/root/notify.sh' 3) oc create configmap mycustomcheck --from-file=/root/ 4) oc volume dc/ipf --add --overwrite --name=config-volume --mount-path=/root --source='{"configMap": { "name": "mycustomcheck"}}' 5) the ipfailover can work for now and check the check.sh have the right oc rsh $pod /root/check.sh 6) scale the router pod to 0 7) the VIP will be deleted 8) scale the router pod to 1 9) the VIP will not be rollback since the check.sh will be 'permission denied' oc rsh ipf-4-0qv8c /root/check.sh rpc error: code = 2 desc = oci runtime error: exec failed: exec: "/root/check.sh": permission denied
Please try oc rsh ipf-4-0qv8c ls -l /root oc rsh ipf-4-0qv8c cat keepalived/lib/config-generators.sh
hi, Phi here is the file: http://pastebin.test.redhat.com/448225
zzhao@redhat.com The script does a chmod +x on the files. This works for me but it doesn't seem to work for your test. I would like to understand this better. Would it be possible to look around on your test machine? If not please try: mount ls -l /root echo "ls" > /root/ppp chmod +x /root/ppp ls -l /root /root/ppp This will verify that the files exist with the correct permission, we can change /root and files do execute. Thanks, phil
@Phil Cameron please notice the following step 6-9 reproduce steps: 1) create two files /root/check.sh and /root/notify.sh cat check.sh </dev/tcp/172.30.157.37/80 ##router service ip/port 2) Create ipfailover pod oadm ipfailover ipf --create --virtual-ips='172.18.3.100' --replicas=1 -w 80 --check-interval=4 --check-script='/root/check.sh' --latest-images --notify-script='/root/notify.sh' 3) oc create configmap mycustomcheck --from-file=/root/ 4) oc volume dc/ipf --add --overwrite --name=config-volume --mount-path=/root --source='{"configMap": { "name": "mycustomcheck"}}' 5) the ipfailover can work for now and check the check.sh have the right oc rsh $pod /root/check.sh 6) scale the router pod to 0 7) the VIP will be deleted 8) scale the router pod to 1 9) the VIP will not be rollback since the check.sh will be 'permission denied' oc rsh ipf-4-0qv8c /root/check.sh rpc error: code = 2 desc = oci runtime error: exec failed: exec: "/root/check.sh": permission denied
The ipfailover dc needs to have defaultMode: 493 volumes: - configMap: defaultMode: 493 <<<< add this line. name: mycustomcheck name: config-volume The change makes the scripts executable. The previous code change is not needed. However, the docs need to be fixed. I will fix the documentation and remove the previous fix for this. I'll let you know the PRs when it is done.
holy mess, 493? should we be using octal? ?0755? Does that not work? decimal modes are for the birds...
0755 works and that is what we should doc. But we need to warn people that when it comes back from the server it will be converted to decimal.
origin PR 12829 removes the unneeded chmod +x openshift-docs PR 3680 documents how to make the script files executable
Commit pushed to master at https://github.com/openshift/origin https://github.com/openshift/origin/commit/7ee93779891ec49b416f552e9d202923ac2e987c 1408172 - ipfailover - `Permission denied for check and notify scripts Reverted the last changes. chmod +x is not correct. The fix is in adding volumes:configMap:defaultMode: 493 in the RC bug 1408172 https://bugzilla.redhat.com/show_bug.cgi?id=1408172
@Phil Cameron https://github.com/openshift/openshift-docs/pull/3680/files#diff-a61f10a803ea6517400a5aea7f6cf169R360 Please correct it to '493' instead of '0755' in json format.
zzhao@redhat.com Fixed #15
This has been merged into ocp and is in OCP v3.5.0.18 or newer.
Verified this bug using the following command to mount the script: 'oc volume dc/ipf --add --overwrite --name=config-volume --mount-path=/root --source='{"configMap": { "name": "mycustomcheck", "defaultMode": 493}}'
Commit pushed to master at https://github.com/openshift/openshift-docs https://github.com/openshift/openshift-docs/commit/fa7a745db3e74feb7fcdd1fbccadca95db436389 ipfailover - Permission denied for check and notify scripts Document setting execute permission in configMap in the RC volumes:configMap:defaultMode: 493 bug 1408172 https://bugzilla.redhat.com/show_bug.cgi?id=1408172
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:0884