Bug 1497035
| Summary: | [osp12]memcached container doesn't have logs for debug | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Artem Hrechanychenko <ahrechan> |
| Component: | openstack-containers | Assignee: | Dan Prince <dprince> |
| Status: | CLOSED ERRATA | QA Contact: | Omri Hochman <ohochman> |
| Severity: | low | Docs Contact: | Andrew Burden <aburden> |
| Priority: | low | ||
| Version: | 12.0 (Pike) | CC: | dprince, lruzicka, m.andre, rhallise, sasha |
| Target Milestone: | rc | Keywords: | Triaged |
| Target Release: | 12.0 (Pike) | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-12-13 19:15:14 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
According to Alex in the upstream bug report [1] this is not specific to containers and we can observe the same behavior with memcached running on BM. It appears we do not pass the '-v' option to the memcached command which cause it to not write any logs. I'd still keep this bug open and in the DFG:Containers since I observed /var/log/memcached.log is created as a directory on the host, and the file is not bind mounted in the /var/log/containers directory so there is still some work for us to do. [1] https://bugs.launchpad.net/tripleo/+bug/1720183 [heat-admin@overcloud-controller-2 ~]$ sudo cat /var/log/containers/memcached/memcached.log [heat-admin@overcloud-controller-2 ~]$ sudo docker logs memcached [heat-admin@overcloud-controller-2 ~]$ Is it normal that we still have no logs for memcached? Artem, I checked in the environment and all the patches are in there. Maybe your memcached service is not being used and has nothing to log? Verified with (OSP12 RC)
memcached-1.4.39-2.el7ost.x86_64
openstack-tripleo-heat-templates-7.0.3-16.el7ost.noarch
paunch debug --file /var/lib/tripleo-config/hashed-docker-container-startup-config-step_1.json --container memcached --action print-cmd
docker run --name memcached-kxdb660m --detach=true --net=host --privileged=false --restart=always --volume=/etc/hosts:/etc/hosts:ro --volume=/etc/localtime:/etc/localtime:ro -
ppet:/etc/puppet:ro --volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro --volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro --volume=
rts/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro --volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro --volume=/dev/log:/dev/log --volume=/etc/ssh/ssh_know
h/ssh_known_hosts:ro --volume=/var/lib/config-data/memcached/etc/sysconfig/memcached:/etc/sysconfig/memcached:ro --volume=/var/log/containers/memcached:/var/log/ 192.168.0.1:8
nstack-memcached-docker:12.0-20171128.1 /bin/bash -c source /etc/sysconfig/memcached; /usr/bin/memcached -p ${PORT} -u ${USER} -m ${CACHESIZE} -c ${MAXCONN} $OPTIONS >> /var/l
g 2>&1
now I see the file exist under /var/log/containers/memcached/memcached.log
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:3457 |
Description of problem: Replace controller scenario Stack update stuck on memcached container deployment on controller-3 But we cannot get any information from memcached [heat-admin@controller-2 ~]$ sudo docker inspect memcached |grep command "config_data": "{\"start_order\": 1, \"image\": \"192.168.24.1:8787/rhosp12/openstack-memcached-docker:2017-09-27.3\", \"environment\": [\"TRIPLEO_CONFIG_HASH=010a1638903ff5c9a080e00eeb23cc0f\"], \"command\": [\"/bin/bash\", \"-c\", \"source /etc/sysconfig/memcached; /usr/bin/memcached -p ${PORT} -u ${USER} -m ${CACHESIZE} -c ${MAXCONN} $OPTIONS\"], \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/puppet:/etc/puppet:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\", \"/sys/fs/selinux:/sys/fs/selinux\", \"/var/lib/config-data/memcached/etc/sysconfig/memcached:/etc/sysconfig/memcached:ro\"], \"net\": \"host\", \"privileged\": false, \"restart\": \"always\"}", [heat-admin@controller-2 ~]$ sudo docker exec -it memcached /bin/bash tput: No value for $TERM and no -T specified tput: No value for $TERM and no -T specified bash: hostname: command not found ()[memcached@ /]$ ps PID TTY TIME CMD 72 ? 00:00:00 bash 84 ? 00:00:00 ps tput: No value for $TERM and no -T specified tput: No value for $TERM and no -T specified bash: hostname: command not found ()[memcached@ /]$ ps -aux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND memcach+ 1 0.0 0.0 11636 1376 ? Ss 13:20 0:00 /bin/bash -c source /etc/sysconfig/memcached; /usr/bin/memcached -p ${PORT} -u ${USER} -m ${CACHESIZE} -c ${MAXCONN} $OPTIONS memcach+ 7 0.0 0.0 629048 2384 ? Sl 13:20 0:00 /usr/bin/memcached -p 11211 -u memcached -m 7941 -c 8192 -l 172.17.1.23 -U 11211 -t 8 >> /var/log/memcached.log 2>&1 memcach+ 72 0.5 0.0 11640 1708 ? Ss 14:57 0:00 /bin/bash memcach+ 91 0.0 0.0 47448 1664 ? R+ 14:57 0:00 ps -aux tput: No value for $TERM and no -T specified tput: No value for $TERM and no -T specified bash: hostname: command not found ()[memcached@ /]$ cat /var/log/memcached.log cat: /var/log/memcached.log: No such file or directory tput: No value for $TERM and no -T specified tput: No value for $TERM and no -T specified bash: hostname: command not found Version-Release number of selected component (if applicable): docker images - 2017-09-27.3 openstack-keystone-12.0.1-0.20170907172639.6a67918.el7ost.noarch python-openstackclient-lang-3.12.0-0.20170821150739.f67ebce.el7ost.noarch openstack-ironic-inspector-6.0.1-0.20170920142417.77e2b1a.el7ost.noarch openstack-puppet-modules-11.0.0-0.20170828113154.el7ost.noarch openstack-tripleo-common-7.6.1-0.20170922213744.el7ost.noarch openstack-mistral-common-5.1.1-0.20170925134000.d1a917e.el7ost.noarch openstack-nova-api-16.0.1-0.20170921091002.edd59ae.el7ost.noarch openstack-nova-conductor-16.0.1-0.20170921091002.edd59ae.el7ost.noarch openstack-glance-15.0.1-0.20170920231317.9091d26.el7ost.noarch openstack-nova-compute-16.0.1-0.20170921091002.edd59ae.el7ost.noarch openstack-heat-api-9.0.1-0.20170925083032.317b841.el7ost.noarch openstack-swift-object-2.15.2-0.20170913222602.34601cc.el7ost.noarch openstack-swift-proxy-2.15.2-0.20170913222602.34601cc.el7ost.noarch openstack-ironic-api-9.1.1-0.20170921133213.96f2438.el7ost.noarch puppet-openstack_extras-11.3.1-0.20170906070209.b99c3a4.el7ost.noarch openstack-mistral-engine-5.1.1-0.20170925134000.d1a917e.el7ost.noarch openstack-tripleo-puppet-elements-7.0.0-0.20170914203705.2094778.el7ost.noarch openstack-tripleo-common-containers-7.6.1-0.20170922213744.el7ost.noarch openstack-nova-common-16.0.1-0.20170921091002.edd59ae.el7ost.noarch openstack-tripleo-heat-templates-7.0.1-0.20170925173113.afbe64a.el7ost.noarch openstack-swift-container-2.15.2-0.20170913222602.34601cc.el7ost.noarch openstack-neutron-common-11.0.1-0.20170923193224.5b0191f.el7ost.noarch openstack-neutron-11.0.1-0.20170923193224.5b0191f.el7ost.noarch openstack-neutron-openvswitch-11.0.1-0.20170923193224.5b0191f.el7ost.noarch openstack-heat-engine-9.0.1-0.20170925083032.317b841.el7ost.noarch openstack-ironic-conductor-9.1.1-0.20170921133213.96f2438.el7ost.noarch openstack-tempest-17.0.0-0.20170901201711.ad75393.el7ost.noarch openstack-mistral-executor-5.1.1-0.20170925134000.d1a917e.el7ost.noarch openstack-tripleo-validations-7.4.0-0.20170918153806.efe8a72.el7ost.noarch openstack-selinux-0.8.10-0.20170922165741.52b3fe8.el7ost.noarch openstack-nova-placement-api-16.0.1-0.20170921091002.edd59ae.el7ost.noarch openstack-swift-account-2.15.2-0.20170913222602.34601cc.el7ost.noarch python-openstacksdk-0.9.17-0.20170821143340.7946243.el7ost.noarch python-openstackclient-3.12.0-0.20170821150739.f67ebce.el7ost.noarch openstack-heat-common-9.0.1-0.20170925083032.317b841.el7ost.noarch puppet-openstacklib-11.3.1-0.20170921022915.6e2b844.el7ost.noarch openstack-tripleo-ui-7.4.1-0.20170925201442.2bb5965.el7ost.noarch openstack-ironic-common-9.1.1-0.20170921133213.96f2438.el7ost.noarch openstack-mistral-api-5.1.1-0.20170925134000.d1a917e.el7ost.noarch openstack-nova-scheduler-16.0.1-0.20170921091002.edd59ae.el7ost.noarch openstack-neutron-ml2-11.0.1-0.20170923193224.5b0191f.el7ost.noarch openstack-heat-api-cfn-9.0.1-0.20170925083032.317b841.el7ost.noarch openstack-tripleo-image-elements-7.0.0-0.20170914203419.526772d.el7ost.noarch openstack-zaqar-5.0.1-0.20170905222047.el7ost.noarch How reproducible: Always Steps to Reproduce: 1.Deploy OSP12 OC with 3ctrl+1comp+ 1 empty ironic node 2.https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/11/html-single/director_installation_and_usage/#sect-Replacing_Controller_Nodes 3. steps 5-8 http://etherpad.corp.redhat.com/JtZ84Hp2nQ Actual results: memcached container without logs Expected results: Additional info: